This paper presents the SmartBugs, an easy-to-use extensible execution framework that simplifies the execution of analysis tools on solidity smart contracts and facilitates reproducibility. It supports 10 tools and two datasets, one used to evaluate the precision of analysis tools (143 annotated vulnerable contracts with 208 tagged vulnerabilities, divided into 10 categories), another contains 47,518 unique contracts collected through Etherscan.

SmartBugs is open-source: https://smartbugs.github.io

Five main parts:

the command-line interface/a web-based user interface
the tool configurations, 
the Docker images of the tools, 
the datasets of smart contracts,  
the SmartBugs Runner, which brings all the parts together to execute the analysis tools.

Requirements:

Docker,
Python3 (PyYAML, solidity_parser)

10 tools:

HoneyBadger,
Maian, 
Manticore, 
Mythril, 
Osiris, 
Oyente, 
Securify, 
Slither, 
SmartCheck, 
Solhint.

LoC: number of lines of code on smart contracts for each category. Other: uninitialized data and the possibility of locking down Ether.

each tool docker image should have a configuration file (.YAML file)

docker_image:
    default: qspprotocol/securify-usolc 
    solc<5: qspprotocol/securify-0.4.25 
cmd: --livestatusfile /results/output.json -fs
output_in_files:
    folder: /results/output.json

named datasets:

reentrancy: dataset/reentrancy arithmetic:

- dataset/arithmetic

- dataset/reentrancy/reentrance.sol

command-line interface:

smartBugs.py [-h, --help]
(--file FILES | --dataset DATASETS)
--tool TOOLS --info TOOLS
--skip-existing --processes PROCESSES
--list {tools, datasets}

Usage Example: To run the tools Oyente and Mythril against the contracts in the named dataset reentrancy

smartBugs.py --tool oyente mythril --dataset reentrancy

To analyze a specific file (or folder), we can use the option --file

python3 smartBugs.py --tool all --file dataset/reentrancy/simple_dao.sol