Frameworks and other Web-related software used in scientific computing, cloud computing and high-performance computing.

• CRAWLERS
• BROWSER
• JAVASCRIPT
• Network
  • FTP
• TEMPLATES
• WEB-FRAMEWORKS
  • Django
• WEB-SERVER
  • Security
  • Utilities

---

CRAWLERS

• scrapy :: A fast high-level web crawling and screen scraping framework for Python. Source code.
  • scrapy-cluster :: This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster. Documentation.
• Soupy :: A wrapper around BeautifulSoup that makes it easier to build complex queries when wrangling web data.

Resources

• image_project :: A group learning project to scrape flickr, extract some GPS EXIF info from the images, save the images to local storage, and insert some meta data into an sqlite database using SQLAlchemy.
• Python Web Scraping Tutorial using BeautifulSoup.

---

BROWSER

• Air :: An ultra-lightweight static site generator.
• Brython :: a.k.a. Browser Python is an implementation of Python 3 running in the browser.
• Ghost.py :: Webkit based scriptable web browser for python. Read the Documentation.
• pyotherside :: Asynchronous Python-3 Bindings for Qt-5. Source code
• Python-Progressbar :: A text progress bar library for Python. Documentation.

---

JAVASCRIPT

• jiphy :: Your client side done in a jiphy. Python to JavaScript 2-way converter.

---

Network

• RxPY :: A library for composing asynchronous and event-based programs using observable collections and LINQ-style query operators in Python.

FTP

• pyftpdlib :: Extremely fast and scalable Python FTP server library.

---

TEMPLATES

• cookiecutter-tapioca :: A cookiecutter template for new tapioca wrappers.

---

WEB FRAMEWORKS

• Falcon :: A high-performance Python framework for building cloud APIs and web app backends.
• palladium :: A pluggable framework for easily setting up predictive analytics services as web services and developing real-world machine learning solutions.
• redactvideo :: A Web application for redacting and publishing police body worn video as efficiently as possible.
• Spyre : A Web Application Framework for providing a simple user interface for Python data projects and data visualization, so you'll also need pandas and matplotlib in addition to cherrypy, with jinja2 templating.
• Tornado :: A Python web framework and asynchronous networking library, originally developed at FriendFeed. Source code.

Django

• The Django-project.
• django-adminplus :: Easily add custom views to the Django admin.
• django-mailgun :: A Django email backend for Mailgun.

Resources

• Hello Web App :: An Introduction to web app development using Python and Django with notes on how to package and install necessary files for deployment on Heroku and the code repository for Hello Web App.

---

WEB-SERVER

• Just-Metadata :: A tool that gathers and analyzes metadata about IP addresses. It attempts to find relationships between systems within a large dataset. https://www.christophertruncer.com/just-metadata-intel-gathering-and-analysis-of-ip-metadata/
• python-webpack :: Python bindings to webpack.
• restcommander :: Fast Parallel Async HTTP client as a Service to monitor and manage 10,000 web servers.
• thriftpy :: Pure python implementation of Apache Thrift. Documentation.
• werkzeug :: A flexible WSGI implementation and toolkit. Source code.

Security

• PenTestScripts :: Scripts that are useful for pen tests.
• PBP :: A simple python wrapper and a command line interface around libsodium, to provide basic functionality resembling PGP.
• Simple-RBAC :: A simple role based access control utility for Python.
• SQLMap :: Automatic SQL injection and database takeover tool. Source code on github.

Utilities

• confix :: A language-agnostic configuration parser for Python.
• EyeWitness :: This utility is designed to take screenshots of websites, RDP services, and open VNC servers, provide some server header info, and identify default credentials if possible.