Weka (Waikato Environment for Knowledge Analysis) has a component called the WEKA Experimenter to run large amounts of experiments.
OpenML can be used right from the WEKA Experimenter by selecting it as the result destination. The current beta integration is available as a seperate WEKA version which can be downloaded here:
Open WEKA allows you to run OpenML Tasks in the Weka Experimenter. You can solve OpenML tasks locally and/or automatically upload your experiments to OpenML.
Create an account on OpenML.org (click the user icon in the top bar). You need this only if you want to upload your results.
Download the extended WEKA environment by clicking on the Download button above. Open the jar file.
After starting Weka, choose the 'Experimenter' from the GUI Chooser.
In the Weka Experimenter, click the "New" button. (For the moment this is the only option, a more dedicated GUI will be released in the near future.)
If you want to upload experiments to OpenML.org, choose 'OpenML.org' under 'Results Destination'. Connect to your OpenML account using the 'Login' button.
The Experiment Type should now be "OpenML Task". All other experimenter inputs should be disabled (they are defined in OpenML tasks).
In the "Tasks" panel, click the "Add New" button to add new Tasks. Insert the task id's as comma-separated values (e.g., '1,2,3,4,5'). Use this form to search for interesting tasks. In the future this will also be integrated in WEKA.
Add algorithms in the "Algorithm" panel.
Go to the "Run" tab, and click on the "Start" button.
The experiment will be executed, and if indicated, also sent to OpenML.org. When the experiment is finished, the results can be inspected in the "Analyse" tab.
In your browser, log in to OpenML.org. Click on your name and choose 'My runs' to see a list of all submitted runs. They can now be queried together with all OpenML results. More overviews of your personal experiments will be added soon.
Warning Please note that this is a beta version, which is under (very) active development. Please report any bugs that you may encounter to email@example.com. Whenever a result is uploaded, it can only be inspected by a query on the Web interface. At the moment, we are integrating this into our Weka package. Not all algorithms have been integrated yet, we are also working on this. A new major release is planned on June 14th, 2013.
Several R packages for running machine learning experiments have OpenML support. They are currently under development.
You can design OpenML workflows in KNIME to directly interact with OpenML. The KNIME plugin is currently under development.
You can design OpenML workflows in RapidMiner to directly interact with OpenML. The RapidMiner plugin is currently under development.