I/O Streams
This page explains Capy’s I/O stream concepts for generic data transfer operations. Understanding the six concepts and their buffer ownership patterns is key to writing efficient, composable network code.
Code snippets assume using namespace boost::capy; is in effect.
|
Part I: Overview and Philosophy
Capy provides six I/O concepts organized into three levels of abstraction, each with distinct buffer ownership semantics.
The Three Levels
| Level | Read Side | Write Side | Characteristics |
|---|---|---|---|
Partial |
|
|
Single operation, may transfer less than requested |
Complete |
|
|
Loops internally, fills/drains buffers completely, EOF signaling |
Buffer-Owned |
|
|
Callee owns buffers, enables zero-copy transfers |
Buffer Ownership Patterns
The concepts divide into two ownership models:
- Caller-Owns-Buffers (ReadStream, WriteStream, ReadSource, WriteSink)
-
The caller provides buffer memory. The I/O operation reads into or writes from the caller’s buffers. This is the traditional model matching OS system calls.
- Callee-Owns-Buffers (BufferSource, BufferSink)
-
The I/O object owns the buffer memory. BufferSource provides
const_bufferdescriptors pointing to its internal storage. BufferSink provides writablemutable_bufferdescriptors. This enables zero-copy data flow.
Concept Relationships
read_some] WS[WriteStream
write_some] RSrc[ReadSource
read] WSink[WriteSink
write + EOF] end subgraph calleeOwns [Callee Owns Buffers] BSrc[BufferSource
pull] BSink[BufferSink
prepare + commit] end RS -.->|loops to fill| RSrc WS -.->|loops to drain| WSink BSrc -->|push_to| WSink BSrc -->|push_to| WS RSrc -->|pull_from| BSink RS -->|pull_from| BSink
The stream concepts (read_some/write_some) map directly to OS system calls.
The source/sink concepts build loops around partial operations. The buffer
concepts invert ownership for zero-copy efficiency.
Part II: Stream Concepts (Partial I/O)
The stream concepts are modeled after Boost.Asio’s AsyncReadStream and
AsyncWriteStream. They provide single-operation semantics that may transfer
fewer bytes than requested.
ReadStream
A ReadStream provides partial read operations via read_some():
template<typename T>
concept ReadStream = requires(T& stream, MutableBufferSequence auto buffers)
{
{ stream.read_some(buffers) } -> IoAwaitable;
// Returns io_result<std::size_t>
};
Behavior
read_some() reads at least one byte (unless EOF or error) but may
read fewer bytes than the buffer can hold:
char buffer[1024];
auto [ec, n] = co_await stream.read_some(mutable_buffer(buffer, 1024));
// n might be 100, even though buffer holds 1024
This matches OS behavior: recv() returns whatever data is currently available.
WriteStream
A WriteStream provides partial write operations via write_some():
template<typename T>
concept WriteStream = requires(T& stream, ConstBufferSequence auto buffers)
{
{ stream.write_some(buffers) } -> IoAwaitable;
// Returns io_result<std::size_t>
};
Behavior
write_some() writes at least one byte (unless error) but may write
fewer bytes than provided:
std::string data = /* 10000 bytes */;
auto [ec, n] = co_await stream.write_some(make_buffer(data));
// n might be 8192, even though data has 10000 bytes
This matches OS behavior: send() may accept only part of your data when
kernel buffers are constrained.
Part III: Source/Sink Concepts (Complete I/O with EOF)
Sources and sinks build on streams by looping internally until the operation completes. They also provide explicit EOF signaling for proper stream termination.
ReadSource
A ReadSource provides complete read operations via read():
template<typename T>
concept ReadSource = requires(T& source, MutableBufferSequence auto buffers)
{
{ source.read(buffers) } -> IoAwaitable;
// Returns io_result<std::size_t>
};
Behavior
read() fills the buffer completely before returning, looping as needed:
char header[16];
auto [ec, n] = co_await source.read(mutable_buffer(header, 16));
// If successful: n == 16 (always fills completely)
WriteSink
A WriteSink provides complete write operations with explicit EOF signaling:
template<typename T>
concept WriteSink = requires(T& sink, ConstBufferSequence auto buffers, bool eof)
{
{ sink.write(buffers) } -> IoAwaitable; // Write all bytes
{ sink.write(buffers, eof) } -> IoAwaitable; // Write + optional EOF
{ sink.write_eof() } -> IoAwaitable; // Signal end of data
};
Behavior
write() sends all bytes before returning:
std::string body = /* 10000 bytes */;
auto [ec, n] = co_await sink.write(make_buffer(body));
// If successful: n == 10000 (always writes completely)
The EOF variants signal that no more data will follow:
// Option 1: write then signal EOF separately
co_await sink.write(make_buffer(data));
co_await sink.write_eof();
// Option 2: combined call (more efficient)
co_await sink.write(make_buffer(data), true);
After successful write_eof() or write(buffers, true), no further writes
are permitted.
Part IV: Buffer Concepts (Callee-Owns-Buffers)
The buffer concepts invert the traditional ownership model. Instead of the caller providing buffers, the I/O object owns the buffer memory and provides descriptors to it. This enables zero-copy data transfer.
BufferSource
A BufferSource produces data by providing const_buffer descriptors to
its internal storage:
template<typename T>
concept BufferSource = requires(T& src, const_buffer* arr, std::size_t max_count)
{
{ src.pull(arr, max_count) } -> IoAwaitable;
// Returns io_result<std::size_t> where size_t is buffer count
};
Behavior
pull() fills the provided array with buffer descriptors:
const_buffer arr[16];
auto [ec, count] = co_await source.pull(arr, 16);
// arr[0..count-1] now contain descriptors to source's internal data
Return Values
-
Data available:
!ec.failed()is true,count > 0 -
Source exhausted:
!ec.failed()is true,count == 0 -
Error:
ec.failed()is true
Buffer Lifetime
The memory referenced by returned buffer descriptors remains valid until the
next call to pull() or until the source is destroyed. The caller must
consume all data before the next pull().
Example
task<> consume_all(BufferSource auto& source)
{
const_buffer arr[16];
for (;;)
{
auto [ec, count] = co_await source.pull(arr, 16);
if (ec.failed())
co_return;
if (count == 0)
break; // Source exhausted
// Process arr[0..count-1]
for (std::size_t i = 0; i < count; ++i)
process(arr[i].data(), arr[i].size());
}
}
Reference: <boost/capy/concept/buffer_source.hpp>
BufferSink
A BufferSink consumes data by providing writable mutable_buffer descriptors
to its internal storage:
template<typename T>
concept BufferSink = requires(T& sink, mutable_buffer* arr, std::size_t max_count,
std::size_t n, bool eof)
{
{ sink.prepare(arr, max_count) } -> std::convertible_to<std::size_t>; // Sync
{ sink.commit(n) } -> IoAwaitable; // Commit n bytes
{ sink.commit(n, eof) } -> IoAwaitable; // Commit + optional EOF
{ sink.commit_eof() } -> IoAwaitable; // Signal end of data
};
Behavior
The sink provides a two-phase protocol:
-
prepare() (synchronous): Get writable buffer descriptors
-
commit() (asynchronous): Finalize written data
mutable_buffer arr[16];
std::size_t count = sink.prepare(arr, 16);
// Write data into arr[0..count-1]
std::size_t written = fill_buffers(arr, count);
auto [ec] = co_await sink.commit(written);
Return Values
-
prepare(): Returns count of buffers filled (0 means no space available)
-
commit(): Returns
(error_code)indicating success or failure
Buffer Lifetime
Buffers returned by prepare() remain valid until the next call to prepare(),
commit(), commit_eof(), or destruction.
Example
task<> produce_data(BufferSink auto& sink, std::string_view data)
{
mutable_buffer arr[16];
std::size_t offset = 0;
while (offset < data.size())
{
std::size_t count = sink.prepare(arr, 16);
if (count == 0)
{
// No space; flush and retry
auto [ec] = co_await sink.commit(0);
if (ec.failed())
co_return;
continue;
}
// Copy data into sink's buffers
std::size_t written = 0;
for (std::size_t i = 0; i < count && offset < data.size(); ++i)
{
std::size_t chunk = (std::min)(arr[i].size(), data.size() - offset);
std::memcpy(arr[i].data(), data.data() + offset, chunk);
offset += chunk;
written += chunk;
}
auto [ec] = co_await sink.commit(written);
if (ec.failed())
co_return;
}
co_await sink.commit_eof();
}
Reference: <boost/capy/concept/buffer_sink.hpp>
Part V: Transfer Algorithms
Capy provides algorithms that compose the concepts for common transfer patterns.
Composed read() and write()
Free functions build complete operations from partial ones:
#include <boost/capy/read.hpp>
// Read into fixed buffers (loops read_some until full)
auto [ec, n] = co_await read(stream, buffers);
// Read into dynamic buffer (loops until EOF)
auto [ec, n] = co_await read(source, dynamic_buffer);
#include <boost/capy/write.hpp>
// Write all data (loops write_some until empty)
auto [ec, n] = co_await write(stream, buffers);
push_to() - Source Provides Buffers
The push_to function transfers data from a BufferSource to a write destination.
The source owns the buffers, enabling zero-copy when the destination can consume
directly.
#include <boost/capy/io/push_to.hpp>
// BufferSource to WriteSink (complete transfer with EOF)
auto [ec, n] = co_await push_to(buffer_source, write_sink);
// BufferSource to WriteStream (streaming transfer)
auto [ec, n] = co_await push_to(buffer_source, write_stream);
BufferSource → WriteSink: Transfers all data and calls write_eof() on completion.
BufferSource → WriteStream: Transfers all data using write_some() in a loop,
handling partial writes.
Reference: <boost/capy/io/push_to.hpp>
pull_from() - Sink Provides Buffers
The pull_from function transfers data from a read source directly into a
`BufferSink’s internal buffers. The sink owns the buffers, enabling zero-copy.
#include <boost/capy/io/pull_from.hpp>
// ReadSource to BufferSink (complete reads)
auto [ec, n] = co_await pull_from(read_source, buffer_sink);
// ReadStream to BufferSink (streaming with partial reads)
auto [ec, n] = co_await pull_from(read_stream, buffer_sink);
ReadSource → BufferSink: Reads complete buffer fills and commits to sink,
calls commit_eof() on completion.
ReadStream → BufferSink: Uses read_some() for partial reads, committing
data incrementally as it arrives.
Reference: <boost/capy/io/pull_from.hpp>
Choosing the Right Pairing
| Source Type | Sink Type | Function |
|---|---|---|
BufferSource |
WriteSink |
|
BufferSource |
WriteStream |
|
ReadSource |
BufferSink |
|
ReadStream |
BufferSink |
|
Never pair BufferSource with BufferSink directly. Both own their
buffers, so transferring between them requires copying. Use the cross-ownership
patterns above for zero-copy efficiency.
|
Part VI: Type-Erasing Wrappers
Capy provides type-erasing wrappers for each concept, enabling transport-independent
APIs. Your code can accept any_read_stream& and work with any concrete stream
type: TCP sockets, TLS streams, mock streams for testing, or custom transports.
Available Wrappers
| Wrapper | Wraps |
|---|---|
|
Any |
|
Any |
|
Any type satisfying both |
|
Any |
|
Any |
|
Any |
|
Any |
Characteristics
- Reference Semantics
-
Wrappers hold a reference to the wrapped object without taking ownership. The wrapped object must outlive the wrapper.
- Frame Preallocation
-
Constructors preallocate the internal coroutine frame. This reserves memory at server startup so usage can be measured up front, rather than allocating piecemeal as traffic arrives.
- Move-Only
-
Wrappers are non-copyable because the cached frame is per-instance. Move operations transfer the frame ownership.
Example
#include <boost/capy/io/any_stream.hpp>
task<> echo(any_stream& stream)
{
char buffer[4096];
for (;;)
{
auto [ec, n] = co_await stream.read_some(make_buffer(buffer));
if (ec == cond::eof)
break;
if (ec.failed())
co_return;
auto [wec, wn] = co_await write(stream, make_buffer(buffer, n));
if (wec.failed())
co_return;
}
}
// Works with any bidirectional stream
task<> handle_tcp(tcp_socket& sock)
{
any_stream stream(sock);
co_await echo(stream);
}
task<> handle_tls(tls_stream& tls)
{
any_stream stream(tls);
co_await echo(stream);
}
Part VII: The owning<> Wrapper
The type-erasing wrappers have reference semantics. When you need a wrapper
that owns its wrapped object, use the owning<> template.
Usage
#include <boost/capy/io/owning.hpp>
// Create an owning wrapper
owning<any_buffer_source, my_source> src(constructor_arg1, constructor_arg2);
// src IS-A any_buffer_source
any_buffer_source& ref = src;
// Access the owned object
my_source& s = src.get();
Characteristics
- Inherits from Base
-
owning<Base, T>inherits fromBase, so it can be used anywhereBase&is accepted. - Owns Instance of T
-
The wrapper constructs and owns an instance of
T. Constructor arguments are forwarded to `T’s constructor. - Move Semantics
-
Move operations transfer both the owned object and the type eraser state. The base is automatically rebound to the new object location.
- Non-Copyable
-
Like the base wrappers,
owning<>is non-copyable.
Example
// Factory function returning owned source
owning<any_buffer_source, file_source> make_file_source(std::string_view path)
{
return owning<any_buffer_source, file_source>(path);
}
task<> process_file(std::string_view path)
{
auto src = make_file_source(path);
// Use as any_buffer_source
auto [ec, n] = co_await push_to(src, some_sink);
}
Reference: <boost/capy/io/owning.hpp>
Part VIII: Practical Patterns
Choosing the Right Abstraction
| Use Case | Recommended Concept |
|---|---|
Process data as it arrives |
|
Read exactly N bytes before proceeding |
|
Send complete message with termination |
|
Zero-copy from data producer |
|
Zero-copy into data consumer |
|
Transport-independent API |
Type-erasing wrappers ( |
Pattern: Echo Server
task<> echo_server(any_stream& stream)
{
char buffer[4096];
for (;;)
{
// Partial read: get whatever is available
auto [ec, n] = co_await stream.read_some(make_buffer(buffer));
if (ec == cond::eof)
break;
if (ec.failed())
co_return;
// Complete write: send all bytes
auto [wec, _] = co_await write(stream, make_buffer(buffer, n));
if (wec.failed())
co_return;
}
}
Pattern: HTTP Body Transfer
task<> send_response_body(any_write_sink& body, std::string_view content)
{
// Write all content and signal EOF
auto [ec, n] = co_await body.write(make_buffer(content), true);
if (ec.failed())
{
// Handle error
}
}
Pattern: Processing Chain
Data can flow through a chain of transformations, each component implementing the appropriate concepts:
BufferSource] --> Compress[Compressor
BufferSource + WriteSink] Compress --> Encrypt[Encryptor
BufferSource + WriteSink] Encrypt --> Socket[Socket
WriteStream]
Each intermediate transform implements both a source concept (to provide data)
and a sink concept (to receive data). The push_to and pull_from algorithms
connect the chain.
Pattern: Transport-Independent Library
Type-erasing wrappers enable libraries that work with any transport:
// Library code - transport independent
class protocol_handler
{
any_stream& stream_;
public:
explicit protocol_handler(any_stream& s)
: stream_(s)
{
}
task<> run()
{
// Works with any stream type
auto [ec, n] = co_await stream_.read_some(/*...*/);
// ...
}
};
// Application code - concrete transport
task<> handle_client(tcp_socket& sock)
{
any_stream stream(sock);
protocol_handler handler(stream);
co_await handler.run();
}
Summary
| Concept | Method | Behavior | Buffer Owner |
|---|---|---|---|
|
|
Read at least one byte (partial) |
Caller |
|
|
Write at least one byte (partial) |
Caller |
|
|
Fill buffers completely or EOF |
Caller |
|
|
Write all bytes, signal EOF |
Caller |
|
|
Provide buffer descriptors |
Callee (source) |
|
|
Accept data into internal buffers |
Callee (sink) |