In-order to write custom broker we have to extend "org.wso2.carbon.broker.core.BrokerType" and "org.wso2.carbon.broker.core.BrokerTypeFactory" that are available in the jar org.wso2.carbon.broker.core-4.0.7.jar.
Here the implementation of the "org.wso2.carbon.broker.core.BrokerType" will contain the broker logic to that will be used to send and receive events,
and the implementation of "org.wso2.carbon.broker.core.BrokerTypeFactory" will be used as the factory to crate your appropriate BrokerType.
After implementing the broker build that and add that to CEP_HOME/repository/components/lib directory.
Next, you have to create a file called "broker.xml" at wso2cep-2.0.1/repository/conf directory, and add the implantation class as in the following XML:
Note: In the above code snippet change the "org.test.cep.broker.BrokerTypeFactoryImpl" with your implementation class name.
Finally, restart & login to the CEP server, go to Configure -> Broker -> Add, and there you will be able configure your custom broker.
For a detail guide on implementing a Broker Extension follow the guild provided here.