Follow Us:

Goal One Performance Measures, Analysis and Review


Table of Contents >

PERFORMANCE MEASURE 1.1.1.1


PERFORMANCE MEASURE 1.1.1.1 Implement automated position limit alerts for futures, option and swaps markets. FY 2012 – Implement automated position limit monitoring for all commodities under CFTC position limits for the swap market using large trader reporting data.
Lead Program Office: Division of Market Oversight (DMO)
Data Source: Integrated Surveillance System (ISS)
Verification: Internal Review of Output & Beta Testing
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
N/A N/A 100% 100% N/A 100% N/A

Performance Analysis & Review

The performance target was met for FY 2012. Position violation detection engines have been built and are in beta testing for the nine commodities with Federal limits. However, the complexities of granted hedge exemptions, many requiring manual adjustments, make a pure automated violation detection engine close to impossible. The court vacating the imposition of position limits on other commodities has compromised the work product by introducing significant uncertainty into the programming efforts, delaying the final product. Surveillance staff across regions and headquarters, along with the ODT are working together to develop a robust new framework for detection of violations incorporating the existing detection computer code developed by each group. This work is not duplicative but addressed different aspects of problems of limit violation detection. This effort has already been successful in uncovering potential violations.

It is expected that a Federal limit detection engine will be rolled out in production by FY 2013. It will automate the analysis and monitoring of futures-equivalent positions in the various derivative products that are linked to the Core Referenced Futures Contracts (CRFCs) to report overages and potential violations. Using futures-equivalent positions, the solution will process the defined CRFCs and associated position limits, incorporate the trader relationships as defined by the Trader Group reportable entities, and aggregate the positions across entities, apply exemptions, and generate a violations data set. The system will allow users to produce customized reports based on the violations data. It is one of a suite of analytical tools to be used by Commission staff.

PERFORMANCE MEASURE 1.1.1.2


PERFORMANCE MEASURE 1.1.1.2 Implement automated surveillance alerts and a case management system. FY 2012 – Implement automated market profile alerts. Integrate swaps market data into two automated market alerts.
Lead Program Office: Division of Market Oversight (DMO)
Data Source: Integrated Surveillance System (ISS)
Verification: Internal Review of Daily Output from Alerts
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
70%. 100% 100% 100% 100% N/A N/A

Performance Analysis & Review

The performance target was met for FY 2012. The Commission implemented several new automated trading violation detection and referral systems constructed within the SAS environment and utilizing JMP software. Automated surveillance for compliance violations was revitalized and better engineered, leading to a large number of closed case based referrals to DOE. Existing models for alerts have still been maintained or enhanced during the year. It is expected that many of the current Actimize models will be ultimately replaced with more flexible detection engines. For instance, the enhancement, the newly developed basic Profiling (data) Cube completed in FY 2012, is expected to be redeployed within the SAS JMP application by the second quarter of FY 2013.

The planned outcomes for the past period covered by the Strategic Plan were based on an assumption that the Commission would make significant progress in the ownership and control of data. In large part because of staff creativity and resourcefulness, the Commission was able to acquire new datasets enabling refined and substantive surveillance. It must be recognized that alerts and automated surveillance truly can never be completed as markets evolve; however, the initial objectives, alerts, and automated surveillance leading to referrals for compliance violations have been met.

With previous and continuing budget constraints and added surveillance burdens stemming from implementation of new requirements, progress going forward is expected to be extremely curtailed. Yet, the Commission continues to make progress in spite of this resource shortfall, by working with data that currently is available—transactional data and newly acquired source data combined with position data organized within new analytic tools.

The Commission's technology, hardware, software, and intellectual property must keep pace with that of the market. The Commission faces challenges in training and mentoring new and existing staff along with hardware limitations, especially limitations in raw computing power.

As with most technical solutions, the process of acquiring the necessary data, and building the most effective alert programs, is resource intensive. The Commission will continue to work diligently on the remaining and additional alert programs, seeking the most efficient methods of conducting operations. However, staffing levels and budget constraints will continue to be a large obstacle toward success going into FY 2013 and likely into FY 2014.

PERFORMANCE MEASURE 1.1.1.3


PERFORMANCE MEASURE 1.1.1.3 Implement automated trading violation alerts and a case management system. FY 2012 – Implement five automated trading violation alerts.
Lead Program Office: Division of Market Oversight (DMO)
Data Source: Actimize
Verification: Internal Review of Daily Output from Alerts
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
20% 100% 98% 100% 100% 100% 100%

Performance Analysis & Review

The performance target was not met for FY 2012. All staff members are being trained in computer programming and use of new sophisticated software applications. INew analytic methods developed by staff, linked with new source data acquisitions and regular calls for account identification to FCMs and exchanges, have enabled the development of several new means of automated detection. All of the Commission's surveillance staff members have access to detection engines. When the engines are run the results are shared by commodity specialists for evaluation prior to drafting referrals to DOE. The engines are modified by analysts as required by the changing landscape of increasing data quality The automated processes have led to a large number of referrals to DOE for compliance violations and increased efficiencies. It is expected that the single largest compliance and violation issue—wash trade detection—will be automated within the first quarter of FY 2013, freeing up resources to be moved to other surveillance activities. Prior automation initiatives were either incorporated into new processes or abandoned. Surveillance tools were constructed for detection of pre-arranged, wash trading, block trading violations, Federal position limit violations, and general surveillance. The general surveillance tool allows for a thorough and rapid event driven analysis of transactions at the millisecond level. The tool enables the analysis of reportable positions in our large trader system over various time frames for all tenors held by participants.

A case management system was neither evaluated nor implemented during this performance period, but is expected to be evaluated and implemented in the second half of FY 2013.

PERFORMANCE MEASURE 1.1.2.1


PERFORMANCE MEASURE 1.1.2.1 Review information requirements of current and proposed forms. FY 2012 – Implement ownership and control reporting standards for futures and option markets. Implement reportable trader standards for swaps traders.
Lead Program Office: Division of Market Oversight (DMO)
Data Source: Forms
Verification: Final Rule
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
50% 100% 60% 100% N/A N/A N/A

Performance Analysis & Review

The performance target was not met for FY 2012. The Commission completed the notice of proposed rulemaking to implement this objective and as of mid-November, the rulemaking team had: 1) closed the comment period on the proposed rules; 2) completed a summary of all comments received; 3) neared completion of a term sheet; and 4) started consultations with groups within the DMO and other offices and divisions of the Commission for policy, legal, and technology guidance that will be used in drafting the final rules.

Delays in completing the final rules have arisen from a number of sources, including consultation with Commission offices and divisions to improve the cost-benefit analysis in the proposed rules; extensions of the original comment period for the notice of proposed rulemaking; and addressing public comments critical of the proposed rules.

Those assigned to the rule writing team have been instructed to prioritize the project in their CFTC work.

PERFORMANCE MEASURE 1.1.4.1


PERFORMANCE MEASURE 1.1.4.1 Percentage of contracts that are reviewed, in a timely manner, following a finding of market significance, and determined to be in compliance with core principles or referred back to exchange for modification.
Lead Program Office: Division of Market Oversight (DMO)
Data Source: FILAC
Verification: Internal Review of Dates Entered into System; Review of ISS Database to determine significance
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
2% 100% 10% 100% 100% 100% 100%

Performance Analysis & Review

The performance target was not met for FY 2012. During this fiscal year, the Commission completed reviews of 29 certified contracts, three of which were completed within 90 days of the contract meeting specific volume and open interest thresholds for market significance, and thus were considered timely for this performance measure. At the end of FY 2012, there remained 20 contracts that met the volume and open interest thresholds for market significance. These contracts are under review and will be assessed during the FY 2013 performance cycle. In total, Commission staff completed or suspended reviews of 200 other certified futures and option contracts, several of which had been delisted due to a lack of market trading activity and thus required no review. Staff and resource limitations leave the Commission with nearly 3,000 contracts to be reviewed leading into FY 2013. However, the vast majority of the contracts in the backlog may not meet the volume and open interest thresholds for market significance.

The Commission reviews a contract's terms and conditions, and the position limits and accountability standards, to ensure that a contract is not readily susceptible to manipulation and that the position limits or accountability standards are appropriate. Often, a review includes an analysis of the exchange's assumptions regarding supply and demand of the underlying commodity as well as a survey of prevailing cash market practice. In the absence of Commission due diligence, undetected contract flaws or faulty assumptions could lead the contract to be readily susceptible to manipulation, cause disruptions in the cash market, or encourage excessive speculation.

As staff members are released from rule-writing responsibilities, review of contracts that exhibit market significance resumes. The Dodd-Frank Act rulemaking effort continued through FY 2012, and implementation of those rules is expected to impact future targets set in the current Strategic Plan. In response to these external conditions, the Commission modified its targets to more accurately reflect the Commission's capabilities given the existing staffing levels and continued budget constraints. It should be noted, however, that for the contracts currently exhibiting market significance, as reflected in volume and open interest statistics, more than 90 days has elapsed since meeting those thresholds. Thus, clearing the backlog will be a challenge in meeting the goal.

PERFORMANCE MEASURE 1.1.5.1


PERFORMANCE MEASURE 1.1.5.1 Rule submissions are reviewed and a determination is made regarding compliance with the CEA, or referred back to the exchange for correction, as amended by the Dodd-Frank Act and Commission regulations within the required 10-day or 90-day time period.
Lead Program Office: Division of Market Oversight (DMO)
Data Source: FILAC
Verification: Internal Review of Dates Entered into System
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
77% 100% 73% 100% 100% 100% 100%

Performance Analysis & Review

The performance target was not met for FY 2012. The Commission was able to complete reviews of 429 submissions for certification or approval of substantive trading rules, 93 percent of the total submitted throughout the year. In addition, the Commission was also able to complete 36 submissions for certification or approval of substantive product rules, approximately 21 percent of the total on the year. Combining trading and product rule reviews, this is an estimated 73 percent performance rate during the fiscal year. However, limited staffing and resource constraints throughout the fiscal year continued to inhibit the level of reviews of rule amendment filings as planned. It should be noted, however, that Commission staff routinely examine rule amendment filings to ensure compliance with the relevant core principles within the appropriate timeframes. Nevertheless, the completion statistics described above include only those filings for which a formal review has been completed and documented.

It is significant to note, that the backlog of rule amendment certification reviews has grown so much that this goal, as written, may not be met with the existing staff levels. To date, there are 46 submissions for trading rule amendments and 135 submissions for product rule amendments on backlog for review. While the Commission continues to seek the most efficient methods of conducting operations, staffing levels and budget constraints will continue to be the largest obstacle to success going into FY 2013.

PERFORMANCE MEASURE 1.1.6.1


PERFORMANCE MEASURE 1.1.6.1 DCM and SEF applications are reviewed and a determination is made regarding compliance with core principles within statutory time frames.
Lead Program Office: Division of Market Oversight (DMO)
Data Source: Memorandum Initiating Review
Verification: Final Report
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
100% 100% 100% 100% 100% 100% 100%

Performance Analysis & Review

The performance target was met for FY 2012. The Commission received one DCM application on January 31, 2012, and it was designated on September 25, 2012. SEF rules are not final and thus there were no SEF registrations to review.

PERFORMANCE MEASURE 1.2.1.1


PERFORMANCE MEASURE 1.2.1.1 Percentage of major DCMs and SEFs reviewed, during the year. (Structural Sufficiency)
Lead Program Office: Division of Market Oversight (DMO)
Data Source: Memorandum Initiating Review
Verification: Final Report
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
40% 100% 0% 100% 100% 100% 100%

Performance Analysis & Review

The performance target was not met for FY 2012. During this performance cycle, the CFTC completed the vast majority of work in its rule enforcement review (RER) of the disciplinary program at ICE Futures U.S. This RER was completed during the first quarter of FY 2013. Additionally, the Commission initiated a joint RER of the Chicago Board of Trade's (CBOT) and Chicago Mercantile Exchange's (CME) market surveillance program, as well as a market-surveillance RER at ICE Futures U.S. Those reviews are expected to be completed in FY 2013. Although it is expected that there will be significantly more RERs completed in FY 2013 than were completed in the current fiscal year, the ability to meet the performance goal will be limited by SDR and anticipated SEF registration responsibilities.

Due to insufficient resources, the Commission continues to face challenges in prioritizing amongst its rulemaking, SDR registration, DCM oversight, and product review responsibilities. Due to the priority placed on rulemaking and SDR registration, functions outside those activities, including the review of major DCMs, were extremely limited during the fiscal year. As the Commission continues to place a high priority on completing its rulemaking during FY 2013 and the registration of SDRs and SEFs (once the SEF rules are completed as expected in the first or second quarter of FY 2013), challenges in meeting program goals will continue. For example, in an effort to maintain RER examinations, the RER program has been modified so that each examination will on average address a more focused set of DCM program areas (e.g., market surveillance, trade practice surveillance, audit trail, disciplinary, etc.). In this manner, the Commission will attempt to maintain an RER presence at DCMs while working within its limited resources and competing demands.

PERFORMANCE MEASURE 1.2.1.2


PERFORMANCE MEASURE 1.2.1.2 Percentage of non-major DCMs and SEFs reviewed, during the year. (Structural Sufficiency)
Lead Program Office: Division of Market Oversight (DMO)
Data Source: Memorandum Initiating Review
Verification: Final Report
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
20% 100% 13% 100% 100% 100% 100%

Performance Analysis & Review

The performance target was not met for FY 2012. Due to significant rulemaking and SDR registration responsibilities, the CFTC was unable to review the appropriate level of non-major DCMs and SEFs during the performance cycle. The Commission notes, however, of the 13 non-major, non-dormant DCMs during FY 2012, three had no trading activity and two were newly designated. During FY 2012, the Commission completed an RER of NYSE Liffe's trade practice surveillance program. In addition, staff initiated RERs at five non-major DCMs, all of which are expected to be completed in FY 2013. These current RERs include examination of the trade practice surveillance, disciplinary, and audit trail programs at the Minneapolis Grain Exchange; the trade practice surveillance programs at NADEX and OneChicago; the market surveillance program at ELX; and the audit trail program at CBOE Futures Exchange. The Commission is actively pursuing opportunities to improve performance in this area. Although the CFTC expects to complete significantly more RERs of non-major DCMs in FY 2013 than were completed in FY 2012, its ability to meet the performance goal will be limited by SDR and expected SEF registration responsibilities.

The Commission continues to face a challenge of insufficient resources, in addition to rulemaking and SDR registration responsibilities. As a result, functions outside of rulemaking and SDR registration, including the review of non-major DCMs, were extremely limited during the fiscal year. As the Commission remains determined to complete its rulemaking during FY 2013 and is committed to the registration of SDRs and SEFs (once the SEF rules are completed as expected in the first or second quarter of FY 2013), challenges will continue. However, in an effort to complete more RER examinations, the RER program has been modified to focus on a specific area of a DCM's self-regulatory responsibilities, e.g., trade practice surveillance or audit trail, rather than reviewing all or most of a DCM's self-regulatory programs.

PERFORMANCE MEASURE 1.2.2.1


PERFORMANCE MEASURE 1.2.2.1 Percentage of major DCMs and SEFs reviewed, during the year. (Automated Systems and Business Continuity)
Lead Program Office: Division of Market Oversight (DMO)
Data Source: Market Continuity Program
Verification: Final Report
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
80% 100% 80% 100% 100% 100% 100%

Performance Analysis & Review

The performance target was not met for FY 2012. The CFTC conducts system safeguard examinations (SSEs) relying on the expertise of systems risk analysts The Commission completed one SSE for CME Group, which includes four major DCMs (CME, CBOT, NYMEX, and COMEX) during FY 2012. An SSE was not conducted for ICE during FY 2012 due to limited staff resources. The Commission's Market Continuity Program does not have enough systems risk analysts to conduct the targeted level of SSEs for DCMs in addition to other activities: reviewing DCM and SDR applications for designation and registration; assisting with Dodd-Frank Act rulemakings, other automated system-related Commission rulemakings, and policy development; and responding to system disruptions and cyber threats affecting regulated entities.

SSEs involve an assessment of a DCM's compliance with Section 5(d) of the CEA and Core Principle 20, System Safeguards. Each SSE includes a review of some or all of the six principal categories of risk management controls: 1) information security, 2) business continuity and disaster recovery (BC-DR) and pandemic planning, 3) capacity planning processes and testing, 4) computer and network operations, 5) systems development methodology and quality assurance, and 6) physical security and environmental controls.

An examination of CME Group was initiated and an onsite visit was conducted in the fourth quarter FY 2012. Due to examination follow-up activities, the final examination report will be completed during the third quarter of FY 2013. The final report for the examination of ICE, initiated at the end of the fourth quarter of FY 2011 and extended into FY 2012, will be completed in the first quarter of FY 2013.

Regulations for SDR registration were finalized and published in FY 2012, and thus, there were no major SDRs to be reviewed as part of the performance measure. The Commission initiated designation reviews for five SDRs and provisionally designated two SDRs in FY 2012. As part of the designation review process, the Market Continuity Program conducted reviews of SDR applicant compliance with 17 CFR Part 49.24, System Safeguards, on a continuous basis throughout the fiscal year.

Regulations for SEF registration are incomplete and thus, there were no major SEFs to be reviewed as a part of the performance measure.

PERFORMANCE MEASURE 1.2.2.2


PERFORMANCE MEASURE 1.2.2.2 Percentage of non-major DCMs and SEFs reviewed, during the year. (Automated Systems and Business Continuity)
Lead Program Office: Division of Market Oversight (DMO)
Data Source: Market Continuity Program
Verification: Final Report
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
0% 33% 0% 33% 33% 33% 33%

Performance Analysis & Review

The performance target was not met for FY 2012. SSEs were not completed for non-major (Tier 2) DCMs during FY 2012 due to limited staff resources. The Commission notes, however, of the 13 non-major, non-dormant DCMs during FY 2012, three had no trading activity and two were newly designated. The Commission's Market Continuity Program does not have enough systems risk analysts to conduct the targeted level of SSEs for DCMs in addition to other activities: reviewing DCM and SDR applications for designation and registration; assisting with Dodd-Frank Act rulemakings, other automated system-related Commission rulemakings, and policy development; and responding to system disruptions and cyber threats affecting regulated entities.

Examinations of two non-major DCMs , ELX and NYSE Liffe, were initiated and onsite visits were conducted during this performance cycle. Due to follow-up activities, the final examination reports will be completed during the second quarter of FY 2013.

While automated system malfunctions, cyber incidents, or trading interruptions at non-major DCMs do not pose the same degree of risk to the U.S. financial system as similar problems at a major DCM, automated systems and BC-DR resources are critical for any DCM to maintain a comprehensive audit trail, publish timely market data, conduct adequate market and trade practice surveillance, provide large trader reporting, and monitor and enforce compliance with DCM rules and Commission regulations. Non-major DCMs are more likely to experience catastrophic systems failures or security breaches because they often have less capital to invest in systems development, testing, and maintenance, and cannot afford the most current hardware, software, or security measures. Frequently, technology staff at the non-major DCMs have multiple responsibilities across the organization; in many cases, the roles held are in conflict with each other, and do not represent best practices in organizational separation of duties. Examination of non-major DCMs on at least a biennial basis is important to the Commission's regulatory mission.

Regulations for SEF registration are incomplete and thus, there were no minor SEFs to be reviewed as a part of the performance measure.

PERFORMANCE MEASURE 1.3.1.1


PERFORMANCE MEASURE 1.3.1.1 Publish reports for swaps markets activity. FY 2012 – Develop and test aggregation methods to group interest rate swap products.
Lead Program Office: Office of the Chief Economist (OCE)
Data Source: Internal Schedule
Verification: CFTC External Website
ACTUAL
FY 2011
PLAN
FY 2011
ACTUAL
FY 2012
PLAN
FY 2012
PLAN
FY 2013
PLAN
FY 2014
PLAN
FY 2015
N/A N/A 100% 100% 100% 100% 100%

Performance Analysis & Review

The performance target was met for FY 2012. The proposal version of the CFTC Swaps Report has been completed, and launched on the Commission's website on Wednesday, November 14, 2012. The report provides for the general public information about previously opaque OTC swaps similar to that published in the Commitment of Traders Report. The Swaps Report covers multiple asset classes, including interest rates, credit, and commodities. This version of the report will be updated on a weekly basis. The Commission has requested public comment on the proposal, and will accept comments for a period of 30 days following the initial publication. The feedback will help inform the final version of the Swaps Report, which will be published at such time as the data being submitted to SDRs becomes more complete (in terms of both asset classes and entities).

 

< Previous page | Table of Contents | Next Page >