Pipeline

LangBatch provides building blocks to create pipelines for AI inference. You can create a pipeline by combining these components. BatchHandler and BatchDispatcher are the core components of the pipeline.

• Batch Handler

  Handler for the batches

• Batch Dispatcher

  Batching incoming requests