jAER as a live camera server, and Python dataloaders

September 7, 2026 · View on GitHub

This is the user guide for File → Remote and for reading jAER recordings in Python. In-app Show Help on each Remote dialog has filter-specific knobs and protocol details.

jAER does not load TensorFlow or PyTorch. A Python (or OpenCV / ROS) process consumes what jAER publishes.

Pick a path

You want	Open	Consumer
A webcam-like stream for OpenCV, YOLO, Zoom, Meet	OpenCV live camera	`cv2.VideoCapture` or a Linux v4l2 device
Lowest-latency events or frames into a local DNN	DNN shared memory	Python `mmap` + localhost TCP
Frames on a robot or in Foxglove	ROS2 and Foxglove	`ros2 topic` or Foxglove Studio
Train / evaluate offline on recordings	Python dataloaders	`aedat`, `h5py`, or CSV

Same File → Remote sinks work on a live USB camera and on File → Open playback. Playback is a repeatable “camera” for debugging the consumer.

Common steps (any Remote sink)

Select the AEChip that matches the camera (or the recording).
Open a live device, or File → Open a sample / your .aedat4.
File → Remote → the sink you want. That only opens the control window.
Press the green enable toggle at the top of that window. The chip-view overlay must show the sink as running.
Closing the window does not stop publishing. Use the same toggle (red) to stop.

Optional: put a denoiser above the Remote filter in the Filter panel so the consumer sees cleaned events.

OpenCV live camera

File → Remote → OpenCV camera output…

jAER assembles DVS / Davis frames and serves HTTP Motion JPEG. Stock OpenCV treats it as a camera.

import cv2

cap = cv2.VideoCapture("http://127.0.0.1:8090/video.mjpg", cv2.CAP_FFMPEG)
ok, frame = cap.read()

cv::VideoCapture cap("http://127.0.0.1:8090/video.mjpg", cv::CAP_FFMPEG);

Browser preview: http://127.0.0.1:8090/ · still: /snapshot.jpg.

frameSource: Auto / RenderedPixmap follows the chip view (including Davis frames). ApsFrames is intensity only. DvsEventCount is an event histogram (mid-gray = zero).

Linux webcam (Cheese, Zoom, Google Meet): HTTP MJPEG is not a webcam. Check publishV4l2, set outputSize to VGA (640×480), leave v4l2Mjpeg on. Overlay must show /dev/video10 open MJPEG. Load the loopback device once with sudo (modprobe details and Cheese workaround: in-app Help, or scripts/cheese-jaer.sh).

DNN shared memory

File → Remote → DNN shared memory output…

jAER writes a memory-mapped file and a localhost TCP JSON-lines control channel (HELLO on connect, FRAME_READY per buffer). Two payload modes share that path:

`outputMode`	Payload	Typical consumer	TCP
EventCountFrames (These frames can be exposed using ConstantDuration, ConstantCount, or AreaEventCount accumulation in jAER))	64×64 uint8 event-count image	dextra-roshambo-python	`127.0.0.1:14100`
EventWindows	packed `(t, x, y, p)` windows of a constant # of events	rpg_e2vid / FireNet	`127.0.0.1:14101`

Default mmap paths (also shown as mmapPath in the dialog):

Linux/macOS: /tmp/jaer/jaer_dvs_frames.mmap or jaer_dvs_events.mmap
Windows: %TEMP%\jaer\jaer_dvs_frames.mmap or jaer_dvs_events.mmap

EventCountFrames (Roshambo hello world)

outputMode = EventCountFrames. Defaults: 64×64, dvsGrayScale=16, rectifyPolarities=true.
Play a live camera, or Davis346 Roshambo throws with chip Davis346blue.
Enable the filter. Overlay shows the mmap path.
In dextra-roshambo-python:

python consumer.py --jaer-mmap /tmp/jaer/jaer_dvs_frames.mmap --serial_port None --windowed

(--jaer-tcp 127.0.0.1:14100 is implied. --jaer-tcp None polls mmap sequence numbers only.)

If the CNN image is upside-down, toggle flipY (jAER default is lower-left origin; OpenCV / training sets are often top-left).

EventWindows (FireNet / E2VID)

outputMode = EventWindows. Leave eventsPerWindow=0 to use width × height × numEventsPerPixel (0.35, same as E2VID).
Leave flipY on (Python / OpenCV / FireNet use upper-left y).
In rpg_e2vid:

uv run python live_reconstruction.py -c pretrained/E2VID_lightweight.pth.tar --auto_hdr --display --show_events

Byte layout of each mmap slot (double-buffered; seq is the publication fence): in-app Show Help.

ROS2 and Foxglove

File → Remote → ROS2 / Foxglove frame output…

Assembled DVS frames (not the OpenGL pixmap) go to Foxglove Studio and/or ROS2. No ROS2 install is required on the jAER machine (IHMC jros2 / Fast-DDS).

Foxglove

Enable the filter; leave publishFoxglove checked.
Foxglove: Open connection → Foxglove WebSocket → ws://localhost:8765.
Image layout → topic /jaer/event_count (or time-surface / voxel topics).

ROS2 (another machine or the same one with a ROS2 distro): enable publishRos2, then ros2 topic hz /jaer/event_count. Domain ID matches ROS_DOMAIN_ID.

Frame types: EventCountHistogram, TimestampImages, VoxelGrid. Details: in-app Help.

Python dataloaders

jAER records AEDAT-4 (.aedat4) by default — the same DV container iniVation documents. Format table and Save As options: README-file-formats.md.

Y origin: jAER’s chip view has y=0 at the bottom. Files written by DV / iniVation software use OpenCV y=0 at the top. Python image code almost always wants top-left; flip with y = height - 1 - y if the picture is upside-down. File → Save As → DSEC HDF5 stores y=0 at the top.

Easiest numpy: File → Save As

Open the recording (not a live camera). File → Save As… (Ctrl+Shift+S).

DSEC HDF5 (image coordinates, p 0=off / 1=on):

import h5py

with h5py.File("clip.h5", "r") as f:
    t = f["events/t"][:]   # microseconds
    x = f["events/x"][:]
    y = f["events/y"][:]   # y=0 at top
    p = f["events/p"][:]   # 0=off, 1=on

CSV / text (RPG-style default is space-separated t_sec x y p, polarity 0/1):

import numpy as np

t, x, y, p = np.loadtxt("clip.txt", unpack=True)

Optional IN/OUT markers clip the export. Optional Apply EventFilters writes the filtered stream.

Read `.aedat4` directly: `pip install aedat`

aedat 2.2.0 (Neuromorphic Systems) is a small AEDAT-4 decoder. Events are a structured numpy array. Field names are t, x, y, p; on is an alias for p (boolean ON=True).

Tested on sampleData/:

Opens after File → Save As AEDAT-4 (snapshot is a sibling of outInfo, not a child). Current sampleData/ re-exports parse; old zip files with the snapshot inside outInfo still raise RuntimeError: invalid digit found in string.
Events / IMU decode on DVS-only and APS-empty files (DVS128, DVS640, NRV, Steadicam).
Davis APS frames (OPENCV_16U_C1) still raise unknown frame format and stop the iterator. Color DV files such as Davis346redColor MISTLab RoboCup soccer ball approaching from air.aedat4 work (8-bit RGB). Use HDF5/CSV above if you need APS in numpy before aedat supports 16-bit gray.

import aedat
import numpy as np

decoder = aedat.Decoder("recording.aedat4")
print(decoder.id_to_stream())  # stream id → events / frame / imus

chunks = []
for packet in decoder:
    if "events" in packet:
        chunks.append(packet["events"])
    elif "frame" in packet:
        pixels = packet["frame"]["pixels"]  # uint8 (H, W) or (H, W, 3) RGB
events = np.concatenate(chunks) if chunks else np.array([])
# events["t"], events["x"], events["y"], events["p"]  # p also as events["on"]

Fixed-count windows for a PyTorch IterableDataset:

import aedat
import numpy as np
from torch.utils.data import IterableDataset

class Aedat4Windows(IterableDataset):
    def __init__(self, path, events_per_window=8192):
        self.path = path
        self.n = events_per_window

    def __iter__(self):
        decoder = aedat.Decoder(self.path)
        leftover = None
        for packet in decoder:
            if "events" not in packet:
                continue
            ev = packet["events"]
            if leftover is not None:
                ev = np.concatenate([leftover, ev])
                leftover = None
            i = 0
            while i + self.n <= len(ev):
                yield ev[i : i + self.n]
                i += self.n
            leftover = ev[i:] if i < len(ev) else None

# for window in Aedat4Windows("recording.aedat4"):
#     t, x, y, p = window["t"], window["x"], window["y"], window["p"]

Alternatives: dv-processing dv.io.MonoCameraRecording (iniVation; events, frames, IMU); Tonic tonic.io.read_aedat4 (loads the whole event stream).

If jAER offers to create a sibling -rerecord.aedat4 on open (dependent-block LZ4), accept it for faster playback and for Python readers that struggle with that compression.

Doc	What it is
This file	Live Remote sinks + Python reading
In-app Show Help on each File → Remote dialog	Knobs, mmap bytes, v4l2, Foxglove topics
`README-file-formats.md`	Play / record / Save As formats
`README.md`	Install, cameras, sample data
jAER User Guide	Desktop UI
3.4.0 release notes	Screenshots of the Remote dialogs