Integration Frameworks Testing

Integration frameworks testing is a mix of parametric library testing with weblogs for coordinating framework installation (e.g. LLM client frameworks like OpenAI, or DB client frameworks like Postgres).

Running the tests

./build.sh python
./run.sh INTEGRATION_FRAMEWORKS -L python --weblog openai-py@2.0.0 tests/integration_frameworks/llm/openai/test_openai_apm.py -vv

which will run the tests for all frameworks.

Generating cassettes

To generate cassettes for a given framework, run the following command, making sure to include the relevant weblog (e.g. openai-py@2.0.0):

./utils/scripts/generate-integration-framework-cassettes.sh --weblog openai-py@2.0.0

which will run the tests without caring about the test assertions, and will enforce proper API keys to generate the cassettes one time.