When an input to an answer extractor or universal classifier is received that
is longer than the maximum input length that a model can process in a single go, we will, unless chunking is disabled, automatically split that input
up into smaller chunks and will then process each chunk separately.
We use our own semchunk algorithm to chunk inputs
in such a way that they are unlikely to cut off in the middle of an important sentence or paragraph.
Although semchunk is a deterministic algorithm, it
can still be difficult to predict exactly how many chunks will be created for any given input,
due to the fact that the algorithm is designed to create chunks that are semantically meaningful
as possible rather than to create chunks of a fixed size.
The default chunk size is the maximum input length of the model less overhead, which
includes not only boilerplate tokens but also, if a model that takes a query as input is
being used, the number of tokens in the longest statement in that query. For every chunk
that is created, the number of tokens inputted into a model will increase by the number of
tokens in that chunk in addition to the number of boilerplate tokens that are added to the
input.
Additionally, if the chunks are being passed to a model that also takes a statement as
input, such as a universal classifier, that statement will have to be added to each chunk, which will therefore increase the
number of tokens inputted into the model by the number of tokens in the statement
multiplied by the number of chunks. The use of a chunk overlap ratio will also increase
the number of tokens and oftentimes the number of chunks being inputted to a model.