High-Performance Computing in Julia

Build correct numerical programs in Julia, then study the choices that can make them efficient: types and dispatch, memory reuse, reductions, SIMD-friendly loops, real CPU threads, tasks and channels, CPU models of GPU indexing, numerical stencils, and a composed threaded analytics engine. Worked examples establish behavior; hardware execution and performance claims need their own measurements.

11 projects, 275 hands-on levels, run in your browser.

Syllabus

Foundations: code through high-performance computing: Build Julia functions from values, arithmetic, conditions, loops and arrays. Trace small examples, define empty-input behavior, and assemble a numerical report. These foundations prepare you to reason about both correctness and later performance choices.
Type Stability & Performance: Study types, dispatch and inference through typed reductions and generic numerical methods. Distinguish return-type stability from speed, define typed empty identities, and use centered variance rather than subtracting large raw moments. Timing and allocation claims require separate measurements.
Memory & Allocation: Explore dense column-major layout, copy versus view behavior, reusable destinations and broadcast fusion. Observe which storage changes and preserve caller inputs where required. Buffer reuse can reduce allocations without guaranteeing that every operation allocates nothing.
Reductions & Scans: Build folds, scans, chunk reductions and statistics with explicit identities and order. Include an arbitrary initial value once, preserve noncommutative order where required, and recognize that floating regrouping can change results. Centered updates provide a more robust variance calculation.
SIMD & Vectorization: Write numerical loops suitable for compiler optimization, use valid-index assumptions carefully, and evaluate polynomials with multiply-add expressions. Distinguish muladd from guaranteed fused fma semantics. Correct outputs do not demonstrate generated SIMD instructions or a measured speedup.
Multithreading: Use real Julia threads for maps and reductions with disjoint chunk-owned state. Balance nonempty chunks, combine generic seeds once, and merge centered statistics. Integer controls can be exact; floating results may vary with grouping. Concurrency safety also depends on callback and aliasing restrictions.
Parallel Algorithms: Compose scans, stable compaction, sorted-run merging and parallel chunk sorting. Preserve duplicates, output order and promotion across runs. The examples identify which phases actually use threads; the final map-filter-reduce pipeline retains a sequential filter stage.
Tasks & Channels: Coordinate spawned tasks and bounded channels. Preserve result order, wait for all tasks, propagate failures and close streams when downstream work stops. Task completion and deterministic output ordering do not by themselves prevent races or guarantee speed.
GPU-Style Kernels: Model one-based grid and block indexing, guarded accesses, grid-stride coverage, gather/scatter and two-level reductions using ordinary Julia arrays on the CPU. Compose a mapped-array and block-reduction engine. Actual GPU execution, memory synchronization and performance require a separate backend and device validation.
Cache-Friendly Numerics: Implement matrix traversal, tiled multiplication, neighbor stencils and alternative record layouts. Keep boundaries and grid spacing explicit, preserve fractional updates for integer inputs, and distinguish locality reasoning from measured performance. Equivalent floating computations can differ with loop order.
Capstone: A High-Performance Pipeline: Assemble a numeric-column analytics engine with real threaded chunk execution. A reusable query applies a transform, tests eligibility on the transformed value, accumulates count and sum, then computes a survivor-weighted mean. Preserve empty behavior and type promotion, and distinguish this CPU teaching engine from a benchmarked production analytics system.

Key concepts

@inbounds: @inbounds permits bounds-check removal in eligible code. Use it only after proving all accesses valid. Invalid unchecked access has undefined behavior; neither…
Abstract type: A type used to group subtypes, such as Real, that cannot itself be instantiated. Abstract method signatures support generic dispatch. Abstractly typed containe…
Allocation: Reserving storage for a value or buffer. New arrays commonly require heap allocation; the compiler may eliminate some temporary objects. Reusing output storage…
Array of structs: An array of records keeps each record as one element. Concrete isbits records can be stored inline, while other representations may use references. Reading com…
Associativity: The property op(op(a,b),c) == op(a,op(b,c)). It permits regrouping while retaining order. It does not imply commutativity, and ordinary floating addition fails…
Atomic: An atomic operation is indivisible with respect to concurrent operations under its memory semantics. Supported atomic updates can prevent lost updates to a val…
Backpressure: A bounded queue slows a producer when capacity is exhausted. put! waits for space while the channel is open; closure or failure can instead terminate it with a…
Bang convention (!): A trailing ! conventionally warns that a function mutates one or more arguments. It does not specify which argument, enforce mutation, or promise zero allocati…
Block reduction: A two-level model: reduce each block into a partial, then combine the partials. Actual GPU implementations additionally require backend-specific memory and syn…
Boxing: Representing a value through a heap object or reference when its concrete representation is not kept directly. It can add allocation or indirection; whether a…
Branch-free code: A source expression avoiding an explicit conditional, such as finite binary-mask arithmetic. max and clamp may still compile to branches. Machine code and benc…
Broadcast fusion: Combining nested dotted operations into one broadcast computation without materializing their intermediate result arrays. Nondotted calls may break fusion, and…
Broadcasting: Applying a function elementwise with compatible shapes, for example sqrt.(xs). Nested dotted calls normally fuse; a newly returned result array still needs sto…
Cache: Small, fast memory holding copies of data near the CPU. Access order and reuse affect cache behavior, but elapsed time also depends on computation, hardware an…
Channel: A typed queue between tasks. put! on a full open channel and take! on an empty open channel wait. Closing prevents new puts; buffered values remain available b…
Column-major order: Ordinary dense Julia Arrays store the first index contiguously: a matrix is laid out by columns. This often favors a row-index inner loop. Views, sparse arrays…
Concrete type: A fully specified type whose values can be instantiated, such as Int or Float64. isconcretetype distinguishes it from abstract types and incompletely specified…
Discrete Laplacian: The 1D second difference u[i-1]-2u[i]+u[i+1] approximates the second derivative after division by spacing squared. The taught unscaled stencil uses unit spacin…
Dotted assignment: out .= expression broadcasts into an existing destination. With dotted operations it can avoid intermediate result buffers. Shape compatibility, aliasing and c…
eachindex: Returns an index iterator suitable for the supplied array or arrays. It supports different indexing styles, but does not independently guarantee bounds-check e…
eltype: The declared element type of a container. eltype([1.0,2.0]) is Float64. An accumulator seeded with zero(eltype(xs)) has a useful typed identity, but later oper…
fetch: For a Task, fetch waits and returns its result, or propagates task failure. Fetching in input order can preserve output order. Waiting alone cannot fix data ra…
Fold: Applying a binary operation to an accumulator and successive elements. A left fold fixes left-to-right grouping; a generic reduction may regroup. An arbitrary…
Garbage collector: The garbage collector reclaims heap objects that are no longer reachable. Allocation can increase memory pressure and collection work; allocating one object do…
Gather: Reading through an index list: out[k] = xs[idx[k]]. Indices must be valid. Repeated indices repeat reads and outputs; this differs from scatter, where repeated…
Global thread index: For one-based logical block and thread indices, (block-1)*blockdim + thread gives the global position. Other GPU APIs may use zero-based indices; the formula m…
Grid and block: A logical grid contains blocks of logical threads. Whole-block launch sizes can include indices beyond the data, so reads and writes both need guards. The cour…
Grid-stride loop: A logical worker visits tid, tid+total_workers and so on. Positive worker counts and unique starting indices partition a one-based range. CPU simulation of thi…
Horner's method: Evaluating a polynomial by nested multiply-add steps, starting with the highest-degree coefficient. This reduces arithmetic compared with separately forming ev…
In-place operation: Writing into existing caller-provided storage. The Julia ! convention signals mutation of one or more arguments. Reusing the output buffer does not rule out in…
Just-in-time compilation: Julia commonly compiles specialized code on demand and reuses it. First-call time can include inference and compilation; warmed calls may still trigger new com…
Kernel: In a device model, work associated with a logical thread index. This track models index coverage and reductions with ordinary CPU loops over Julia arrays; it d…
Locality: Spatial locality accesses nearby memory; temporal locality reuses data soon. Dense column-major matrices often favor a first-index inner loop. Actual benefit d…
Loop order: The nesting order of loops affects memory traversal and sometimes arithmetic grouping. Mathematically equivalent orders can produce different floating results.…
mapreduce: Transforming each element and combining the results without requiring a separate mapped array. Generic mapreduce may regroup operations. Choose the empty-resul…
Matrix multiply: Matrix multiplication forms each output entry from a dot product of a row and a column. Loop ordering, tiling, arithmetic and optimized libraries all affect sp…
Merge: Combining two sorted sequences using two advancing indices. Equal values must be retained. The output type must accommodate both inputs; merge is also a buildi…
Multiple dispatch: Selecting a method using the types of all positional arguments. Generic methods can specialize on concrete calls; annotations should describe supported inputs…
Multiply-add (muladd): muladd(a,b,c) computes a multiply-add while allowing implementation-dependent fusion and optimization. It does not require one rounding. fma specifies a fused…
nthreads: Threads.nthreads() reports the default thread-pool size; Julia may have other pools. Start Julia with --threads to configure workers. Task migration means thre…
Pairwise summation: Adding values in a balanced tree. It can improve rounding-error bounds over a long sequential sum, but is not more accurate for every input. Built-in sum imple…
Parallel compaction: Counting matches, computing write offsets, and placing retained values into disjoint output ranges. Ordered chunk ranges preserve input order. The taught imple…
Parallel scan: A prefix computation with chunk totals, exclusive offsets of those totals, and local scans. The taught implementation parallelizes local scans; preliminary pha…
Parallel sort: Sorting independent chunks concurrently, then merging their sorted runs. The taught merge rounds are sequential and handle odd leftover runs. Parallel chunk so…
Parametric method: A method or type with parameters, such as f(x::T) where T. Parameters express relationships among types and values. They support generic implementations withou…
Partial results: Results owned by independent chunks and combined after workers finish. Chunk ownership avoids shared accumulator updates. Floating results can vary with chunk…
Permutation: Reordering by a vector containing each source index exactly once. Gathering by that vector preserves all values and their multiplicities. An arbitrary repeated…
Preallocation: Creating a buffer before repeated work and reusing it. This avoids repeatedly allocating that buffer, provided shape and element type remain suitable. Other pa…
Prefix sum: The running sum at each position. Inclusive sums include the current value; exclusive sums stop just before it. Chunked floating scans may differ from sequenti…
Producer-consumer: A producer sends values through a channel and a consumer processes them. Completion, producer failure and downstream cancellation need explicit lifecycle handl…
Race condition: A result depending on unsynchronized access timing. Concurrent read-modify-write on a shared sum can lose updates. Correctly owned chunk partials, appropriate…
Reduction: Combining many values into one result, such as a sum or maximum. Regrouping into chunks requires suitable associativity and identity rules. Floating addition i…
Scan: Returning every prefix result rather than only a final reduction. A chunked scan can compute totals, prefix those totals into offsets, then scan each chunk fro…
Scatter: Writing through an index list: out[idx[k]] = vals[k]. The sequential overwrite model keeps the last write; scatter-add accumulates duplicates. Concurrent dupli…
SIMD: Single Instruction, Multiple Data: one machine instruction processes several values. Julia's @simd permits certain loop transformations; it does not prove…
Slice: For ordinary dense Arrays, indexing a range such as xs[2:4] creates a separate array of selected elements. @view instead shares the parent storage. Copying the…
Specialization: Julia can compile method implementations specialized for argument types. Specialization heuristics and inference determine which versions are generated; generi…
Stencil: An update using neighboring input values. Read from the old state and write a separate new state unless a different update order is intended. Boundary conditio…
Structure of arrays: Structure of arrays stores fields in separate arrays. This can improve locality for a loop reading one field, while requiring coordinated lengths. It is not un…
Task: A schedulable unit of Julia work. Threads.@spawn creates a task eligible for a thread pool; tasks can yield and migrate. A task handle supports waiting, fetchi…
Thread-local accumulation: State belonging to one thread. This differs from chunk-local state, which belongs to a logical work unit that a task may execute on different threads. The taug…
Threads.@spawn: Threads.@spawn schedules a task and returns its handle. Scheduling does not guarantee simultaneous execution or a particular worker. Use structured waiting to…
Threads.@threads: Threads.@threads schedules loop iterations across a thread pool and waits for them. Correctness requires safe memory ownership and synchronization; disjoint de…
Tiling (blocking): Processing subregions to increase reuse before moving on. Tile size and loop order influence cache behavior. Tiling alone does not prove that tiles fit cache,…
Tree reduction: Combining partial results in a tree rather than one long left fold. This exposes parallel work and changes rounding order. Numerical accuracy and runtime depen…
Type inference: The compiler estimates possible value types through a program. Concrete inferred intermediates can remove runtime dispatch, but return-type stability and infer…
Type promotion: Converting mixed operands to compatible types using promotion rules. For example, 1 + 2.0 produces Float64. An Int seed may change type on the first floating a…
Type stability: A function is type-stable when its return type is determined by its argument types, rather than argument values. This often helps inference and specialization;…
Vectorization: Generating machine instructions that process several elements at once. Types, dependencies, aliasing and target hardware affect this optimization. Array-lookin…
View: A window sharing storage with a parent array, commonly made with @view. Writes can affect the parent. A view avoids copying the selected data, but its wrapper…
zero and one: zero(T) and one(T) supply additive and multiplicative identities for supported types. They make empty reductions explicit. A typed seed alone does not guarante…