Skip to content

Cyberfame POC

Summary

This document outlines the Proof of Concept (POC) for Cyberfame, a project aimed at automating the identification, fixing, building, testing, and verification of code changes for security. The POC includes steps for cloning repositories, analyzing dependencies, identifying vulnerabilities, and using multi-agent systems for workflow management.

First Draft Program

  1. Clone Repository:

  2. Clone the repository using Git.

  3. Analyze requirements.txt:

  4. Build a dependency graph.

  5. Identify reused or cyclic dependencies from the graph.

  6. Maven/POM.xml:

  7. Dependency project.

  8. References:

  9. Search for Vulnerabilities:

  10. Check for library vulnerabilities using OSSF and OSV.dev.

  11. Perform web searches or API calls for additional vulnerability information.

  12. Create Vulnerability Table:

  13. Compile a table of libraries and their associated vulnerabilities.

  14. Extra Security Measures:

  15. Use Python along with a linter, SAST, and fuzz-testing for API calls.

Code Evaluation Metrics

  1. Version Pinning:

  2. Pin library versions for reproducibility.

  3. Dependency Graph:

  4. Analyze package.json and requirements.txt or poetry.

  5. Docker SBOM Analysis:

  6. Perform analysis on the Docker SBOM.

Integration and Tools

  • OSV-Scanner and OSV-Scalibr:

  • Integrate for vulnerability scanning.

  • Validation of LLM Outputs:

  • Use existing tools to validate outputs from LLMs for automatic code fixes.
  • Reference: swe-agent and other benchmarks.

Agent Flow - Code Verification

  1. Identify Repository:

  2. Identify the repository to be analyzed.

  3. Create Docker Container:

  4. Create a temporary Docker container for the required framework.

  5. Install Code:

  6. Install the code in the Docker template.

  7. Verify Installation:

  8. Fix dependencies if the installation fails.
  9. Run with provided dependencies.

  10. Check for library updates in dependencies.

  11. Run Test Cases:

  12. Execute available test cases.

  13. Run Sample Program:

  14. Analyze and run a sample program based on Readme.md.

Use Cases and Workflow Mechanisms

  1. Use OSV-Scanner:

  2. Utilize OSV-Scanner for vulnerability detection.

  3. Multi-Agent Systems:

  4. Develop a workflow mechanism using multi-agent systems to identify, fix, build, test, and verify code changes for security.

  5. Third-Party PR Validation:

  6. Validate pull requests for financial systems.

Topics for Building Agent Context

  1. Security Topics:
  2. ZeroDay
  3. CVE
  4. Bug bounties

  5. HaveIBeenPwned

  6. Web Crawlers:

  7. Use web crawlers to get real-time information on changes in the code stack.

Output Examples

  1. Vulnerability Report:
  2. Identify vulnerabilities.
  3. Fix vulnerabilities.

  4. Sandbox code fixes and vulnerability changes.

  5. Self-Hosting LLMs:

  6. Use OpenAI, Anthropic, or Ollama for LLM hosting.

References

  1. OpenVAS
  2. OSV-Scanner
  3. OpenSSF
  4. OSV.dev

Conclusion and Next Steps

This POC outlines the steps and tools required to automate the security verification of code changes. The next steps include implementing the proposed workflows, integrating the necessary tools, and conducting thorough testing to ensure the effectiveness of the solution.

  • simple implementation in python at

https://github.com/slabstech/gaganyatri.in/tree/kassel-bio-hackathon/backend/security-poc