KaiRoWiki - User contributions [en]

Fairengames

2016-03-25T17:21:09Z

KaiRo: /* Rules */

Fair. Play. Profit.

=== Rules ===

* Games should not get into boring repetitiveness.
* If games stop being fun to play, they are not worth playing.
* If a game is a random-powered slot machine for items, say so. Don't let game play be reduced to gambling if it's not the main objective.
* Free players are potential future customers - remind them of paid features but don't annoy them.
* Give free players limited access to paid features from time to time so they experience what the worth of "VIP" etc. actually is.
* Make games fun for a broad spectrum of daily time availability of players (5 min per day, 1h every few days, a few hours per weekend, or hours per day).
* Quality is not optional, it's part of keeping players engaged.
* Stay in dialog with the more engaged players.
* Play your games yourself.
* Provide a convenient bug/task tracker that is open to players.
* Avoid UI pieces that flash to attract attention when there's no immediate non-paid action for players to take there.
* Don't have a whole collection of different currencies for different stores within one game unless there's exchanges and good game stories behind them.
* Leave room for individuality and exploration.

[[Category:Games]]

Fairengames

2016-03-25T17:18:44Z

KaiRo: /* Rules */

Fair. Play. Profit.

=== Rules ===

* Games should not get into boring repetitiveness.
* If games stop being fun to play, they are not worth playing.
* If a game is a random-powered slot machine for items, say so. Don't let game play be reduced to gambling if it's not the main objective.
* Free players are potential future customers - remind them of paid features but don't annoy them.
* Make games fun for a broad spectrum of daily time availability of players (5 min per day, 1h every few days, a few hours per weekend, or hours per day).
* Quality is not optional, it's part of keeping players engaged.
* Stay in dialog with the more engaged players.
* Play your games yourself.
* Provide a convenient bug/task tracker that is open to players.
* Avoid UI pieces that flash to attract attention when there's no immediate non-paid action for players to take there.
* Don't have a whole collection of different currencies for different stores within one game unless there's exchanges and good game stories behind them.
* Leave room for individuality and exploration.

[[Category:Games]]

Fairengames

2016-03-25T17:12:36Z

KaiRo: /* Rules */

Fair. Play. Profit.

=== Rules ===

* Games should not get into boring repetitiveness.
* If games stop being fun to play, they are not worth playing.
* If a game is a random-powered slot machine for items, say so. Don't let game play be reduced to gambling if it's not the main objective.
* Free players are potential future customers - remind them of paid features but don't annoy them.
* Make games fun for a broad spectrum of daily time availability of players (5 min per day, 1h every few days, a few hours per weekend, or hours per day).
* Quality is not optional, it's part of keeping players engaged.
* Stay in dialog with the more engaged players.
* Play your games yourself.
* Provide a convenient bug/task tracker that is open to players.
* Avoid UI pieces that flash to attract attention when there's no immediate non-paid action for players to take there.
* Don't have a whole collection of different currencies for different stores within one game unless there's exchanges and good game stories behind them.

[[Category:Games]]

Fairengames

2016-03-25T16:50:59Z

KaiRo: /* Rules */

Fair. Play. Profit.

=== Rules ===

* Games should not get into boring repetitiveness.
* If games stop being fun to play, they are not worth playing.
* If a game is a random-powered slot machine for items, say so. Don't let game play be reduced to gambling if it's not the main objective.
* Free players are potential future customers - remind them of paid features but don't annoy them.
* Make games fun for a broad spectrum of daily time availability of players (5 min per day, 1h every few days, a few hours per weekend, or hours per day).
* Quality is not optional, it's part of keeping players engaged.
* Stay in dialog with the more engaged players.
* Play your games yourself.
* Provide a convenient bug/task tracker that is open to players.
* Avoid UI pieces that flash to attract attention when there's no immediate non-paid action for players to take there.

[[Category:Games]]

Fairengames

2016-03-25T16:50:19Z

KaiRo:

Fair. Play. Profit.

=== Rules ===

* Games should not get into boring repetitiveness.
* If a game is a random-powered slot machine for items, say so. Don't let game play be reduced to gambling if it's not the main objective.
* Free players are potential future customers - remind them of paid features but don't annoy them.
* Make games fun for a broad spectrum of daily time availability of players (5 min per day, 1h every few days, a few hours per weekend, or hours per day).
* Quality is not optional, it's part of keeping players engaged.
* Stay in dialog with the more engaged players.
* Play your games yourself.
* Provide a convenient bug/task tracker that is open to players.
* Avoid UI pieces that flash to attract attention when there's no immediate non-paid action for players to take there.

[[Category:Games]]

Fairengames

2016-03-25T16:48:38Z

KaiRo:

Fairengames

2016-03-25T16:46:26Z

KaiRo:

Fairengames

2016-03-25T16:43:50Z

KaiRo:

=== Rules ===

* Games should not get into boring repetitiveness.
* If a game is a random-powered slot machine for items, say so. Don't let game play be reduced to gambling if it's not the main objective.
* Free players are potential future customers - remind them of paid features but don't annoy them.
* Make games fun for a broad spectrum of daily time availability of players (5 min per day, 1h every few days, a few hours per weekend, or hours per day).

[[Category:Games]]

Fairengames

2016-03-25T14:46:14Z

KaiRo: Created page with "Category:Games"

[[Category:Games]]

Planet Fox

2014-07-27T17:57:20Z

KaiRo:

The game with the working title "Planet Fox" is an [http://en.wikipedia.org/wiki/Augmented_reality_game augmented reality game] designed to help collection of [https://location.services.mozilla.com/ Mozilla Location Service] (MLS) data. This is a raw concept idea, nothing set in stone or planned to be implemented at this time.

While using elements from the real world like GPS/MLS data and possibly some [http://www.openstreetmap.org/ OpenStreetMap] (OSM) points/tags, the game reality plays on an alien planet.

Basic story line: Players land on the alien planet, collect resources and try to help the Alliance of their choice to conquer this planet.

3 Alliances: Groundfoxes, Fireminers, Spacedusters - prefer ground, underground, skyscraping environments

3 Raw resources: Clay, Ore, Cosmic Particles (correspond directly to [system-wide-unique-per-month] locations, wifi APs, celltowers)
* Maybe we need to go for week instead of month, and we'll need to define the radius of a "unique" location.

Refined to Bricks, Metal, Energy in refining stations (need nice names) that can be built and used by any members of an alliance.
* Factors between raw and refined resouces take away uneven distribution of raw resources.
* Need to be physically in influence area of a refinary of own alliance to convert raw to refined resources (instantly? or defined speed?).
* Refining stations can be built in any place where the raw resource is appearing, but with limited density depending on influence areas.

Buildings can be created, belong to the alliance and make it stronger.
* They need all 3 refined resources to be built (varying amounts for different buildings, more of alliance-own resource).
* Examples are makets (Tower of Commerce, etc.) for "cheaper" trade, buildings increasing refining efficiency in some radius or just strengthening the alliance.
* Locations where buildings can be created are not random, but decided by game leadership, probably based on places that have certain OpenStreetMap tags defined. All players can see places that allow creation of buildings.

Players start collecting raw resources immediately, independent of alliances, cannot do much more than view the world and collect.
* Can we make the game servers ignorant of actual stumbler data and have geolocation servers only tell them the relevant stats?
* They can "buy" themselves into an alliance by donating some amount of raw resources to it (instantly returned in refined form?).
* Refined primary resource can be used to create a transformer unit (name/s?), most actions need those transformer units.
* Contributing to the game lets players advance in technology levels (they start at a basic one).

Interactions:
* 1 transformer unit for building refinary, 1 for destroying refinary of a different alliance.
* Buildings need multiple transformer units from multiple people to build or attack/destroy.
* Shields for refinaries and buildings can be constructed with transformer units.
* Buildings and shields might need at least one transformer unit from a player with high enough technology level, refinaries need basic level only.
* Trade of raw and refined resources is always possible between alliances (player <-> alliance?) at a basic level (fixed rate?). Normally, an alliance will trade away foreign raw resources and own refined resource for foreign refined ones.
* Withdrawal from an alliance requires donating all current resources to the old alliance. Then the player is independent again (keeps tech level) amd needs to earn the basic raw resources to join ("buy into") a different alliance.

Planet Fox

2014-02-19T16:59:31Z

KaiRo: Created page with "The game with the working title "Planet Fox" is an [http://en.wikipedia.org/wiki/Augmented_reality_game augmented reality game] designed to help collection of [https://locatio..."

The game with the working title "Planet Fox" is an [http://en.wikipedia.org/wiki/Augmented_reality_game augmented reality game] designed to help collection of [https://location.services.mozilla.com/ Mozilla Location Service] (MLS) data. This is a raw concept idea, nothing set in stone or planned to be implemented at this time.

While using elements from the real world like GPS/MLS data and possibly some [http://www.openstreetmap.org/ OpenStreetMap] (OSM) points/tags, the game reality plays on an alien planet.

Basic story line: Players land on the alien planet, collect resources and try to help the Alliance of their choice to conquer this planet.

3 Alliances: Groundfoxes, Fireminers, Spacedusters - prefer ground, underground, skyscraping environments

3 Raw resources: Clay, Ore, Cosmic Particles (correspond directly to [system-wide-unique/month] locations, wifi APs, celltowers)
* Maybe we need to go for week instead of month, and we'll need to define the radius of a "unique" location.

Refined to Metal, Bricks, Energy in refining stations (need nice names) that can be built and used by any members of an alliance.
* Factors between raw and refined resouces take away uneven distribution of raw resources.
* Need to be physically in influence area of a refinary of own alliance to convert raw to refined resources (instantly? or defined speed?).
* Refining stations can be built in any place where the raw resource is appearing, but with limited density depending on influence areas.

Buildings can be created, belong to the alliance and make it stronger.
* They need all 3 refined resources to be built (varying amounts for different buildings, more of alliance-own resource).
* Examples are makets (Tower of Commerce, etc.) for "cheaper" trade, buildings increasing refining efficiency in some radius or just strengthening the alliance.
* Locations where buildings can be created are not random, but decided by game leadership, probably based on places that have certain OpenStreetMap tags defined. All players can see places that allow creation of buildings.

Players start collecting raw resources immediately, independent of alliances, cannot do much more than view the world and collect.
* Can we make the game servers ignorant of actual stumbler data and have geolocation servers only tell them the relevant stats?
* They can "buy" themselves into an alliance by donating some amount of raw resources to it (instantly returned in refined form?).
* Refined primary resource can be used to create a transformer unit (name/s?), most actions need those transformer units.
* Contributing to the game lets players advance in technology levels (they start at a basic one).

Interactions:
* 1 transformer unit for building refinary, 1 for destroying refinary of a different alliance.
* Buildings need multiple transformer units from multiple people to build or attack/destroy.
* Shields for refinaries and buildings can be constructed with transformer units.
* Buildings and shields might need at least one transformer unit from a player with high enough technology level, refinaries need basic level only.
* Trade of raw and refined resources is always possible between alliances (player <-> alliance?) at a basic level (fixed rate?). Normally, an alliance will trade away foreign raw resources and own refined resource for foreign refined ones.
* Withdrawal from an alliance requires donating all current resources to the old alliance. Then the player is independent again (keeps tech level) amd needs to earn the basic raw resources to join ("buy into") a different alliance.

Lantea

2013-09-18T23:36:44Z

KaiRo:

[http://stargate.wikia.com/wiki/Lantea Lantea] is a fictional ([http://en.wikipedia.org/wiki/Stargate_Atlantis Stargate: Atlantis]) mostly-ocean planet on which the Ancients erected the artificial "island" city of Atlantis.

The Lantea concept is a web app for mapping and tracking that is targeted to run on [https://wiki.mozilla.org/B2G B2G], but can run anywhere on modern browsers.

Currently, a [http://lantea.kairo.at/ Lantea Maps] web app is hosted at kairo.at and [https://marketplace.mozilla.org/app/lantea-maps/ listed in the Mozilla Marketplace] as well as available for install from [http://www.kairo.at/apps KaiRo.at Apps].

Help making it more awesome - [http://git-public.kairo.at/?p=lantea.git;a=blob;f=README;hb=HEAD README], [http://git-public.kairo.at/?p=lantea.git;a=blob;f=TODO;hb=HEAD TODO], source code: [http://git-public.kairo.at/?p=lantea.git;a=summary kairo.at], [https://github.com/KaiRo-at/lantea GitHub]

Base technologies:
* [http://wwww.openstreetmap.org/ OpenStreetMap] tiles - for rendering maps (should be parameterized so that alternative tiles like Google Maps/Satellite can also be used later on)
* [https://developer.mozilla.org/en/Using_geolocation Geolocation] - esp. watchPosition() for tracking positions
* [https://developer.mozilla.org/en/IndexedDB indexedDB] - for saving settings, current track, etc.
* [https://developer.mozilla.org/en/Using_Application_Cache appCache] - for locally caching resources so the app can be used offline
* [https://developer.mozilla.org/en/HTML/Canvas canvas] - for displaying tiles and painting position/tracks
* [http://en.wikipedia.org/wiki/GPS_eXchange_Format GPX format] - for downloading tracks / storing them externally

Resources:
* [http://concentriclivers.com/slippymap.html Canvas Slippy Map] - a simple canvas-based OpenStreetMap map display
* [https://github.com/dfacts/Slippy-Map-On-Canvas/#readme Slippy Map On Canvas] - a more elaborate canvas-based OpenStreetMap web app that can do e.g. touch events for panning and has zoom buttons
* [http://www.mardy.it/mappero Mappero] (a.k.a. [https://garage.maemo.org/projects/maemo-mapper/ maemo-mapper]) - a Maemo app that solved well what the Lantea app should provide on an HTML base
* [http://wiki.openstreetmap.org/wiki/OsmAnd OsmAnd] - an Android app that is similar

Lantea

2013-09-18T23:34:10Z

KaiRo:

[http://stargate.wikia.com/wiki/Lantea Lantea] is a fictional ([http://en.wikipedia.org/wiki/Stargate_Atlantis Stargate: Atlantis]) mostly-ocean planet on which the Ancients erected the artificial "island" city of Atlantis.

The Lantea concept is a web app for mapping and tracking that is targeted to run on [https://wiki.mozilla.org/B2G B2G], but can run anywhere on modern browsers.

Currently, a [http://lantea.kairo.at/ Lantea Maps] web app is hosted at kairo.at and [https://marketplace.mozilla.org/app/lantea-maps/ listed in the Mozilla Marketplace].

Base technologies:
* [http://wwww.openstreetmap.org/ OpenStreetMap] tiles - for rendering maps (should be parameterized so that alternative tiles like Google Maps/Satellite can also be used later on)
* [https://developer.mozilla.org/en/Using_geolocation Geolocation] - esp. watchPosition() for tracking positions
* [https://developer.mozilla.org/en/IndexedDB indexedDB] - for saving settings, current track, etc.
* [https://developer.mozilla.org/en/Using_Application_Cache appCache] - for locally caching resources so the app can be used offline
* [https://developer.mozilla.org/en/HTML/Canvas canvas] - for displaying tiles and painting position/tracks
* [http://en.wikipedia.org/wiki/GPS_eXchange_Format GPX format] - for downloading tracks / storing them externally

Resources:
* [http://concentriclivers.com/slippymap.html Canvas Slippy Map] - a simple canvas-based OpenStreetMap map display
* [https://github.com/dfacts/Slippy-Map-On-Canvas/#readme Slippy Map On Canvas] - a more elaborate canvas-based OpenStreetMap web app that can do e.g. touch events for panning and has zoom buttons
* [http://www.mardy.it/mappero Mappero] (a.k.a. [https://garage.maemo.org/projects/maemo-mapper/ maemo-mapper]) - a Maemo app that solved well what the Lantea app should provide on an HTML base
* [http://wiki.openstreetmap.org/wiki/OsmAnd OsmAnd] - an Android app that is similar

Lantea

2012-03-31T23:21:44Z

KaiRo:

[http://stargate.wikia.com/wiki/Lantea Lantea] is a fictional ([http://en.wikipedia.org/wiki/Stargate_Atlantis Stargate: Atlantis]) mostly-ocean planet on which the Ancients erected the artificial "island" city of Atlantis.

The Lantea concept is a web app for mapping and tracking that is targeted to run on [https://wiki.mozilla.org/B2G B2G], but can run anywhere on modern browsers.

Currently, a [http://lantea.kairo.at/ Lantea Maps] web app is hosted at kairo.at and listed in the [https://marketplace.mozilla.org/ Mozilla Marketplace] (developer-only for the moment).

Base technologies:
* [http://wwww.openstreetmap.org/ OpenStreetMap] tiles - for rendering maps (should be parameterized so that alternative tiles like Google Maps/Satellite can also be used later on)
* [https://developer.mozilla.org/en/Using_geolocation Geolocation] - esp. watchPosition() for tracking positions
* [https://developer.mozilla.org/en/IndexedDB indexedDB] - for saving settings, current track, etc.
* [https://developer.mozilla.org/en/Using_Application_Cache appCache] - for locally caching resources so the app can be used offline
* [https://developer.mozilla.org/en/HTML/Canvas canvas] - for displaying tiles and painting position/tracks
* [http://en.wikipedia.org/wiki/GPS_eXchange_Format GPX format] - for downloading tracks / storing them externally

Resources:
* [http://concentriclivers.com/slippymap.html Canvas Slippy Map] - a simple canvas-based OpenStreetMap map display
* [https://github.com/dfacts/Slippy-Map-On-Canvas/#readme Slippy Map On Canvas] - a more elaborate canvas-based OpenStreetMap web app that can do e.g. touch events for panning and has zoom buttons
* [http://www.mardy.it/mappero Mappero] (a.k.a. [https://garage.maemo.org/projects/maemo-mapper/ maemo-mapper]) - a Maemo app that solved well what the Lantea app should provide on an HTML base
* [http://wiki.openstreetmap.org/wiki/OsmAnd OsmAnd] - an Android app that is similar

Main Page

2012-03-31T23:19:03Z

KaiRo:

'''This wiki is a random collection of ideas and concepts by Robert Kaiser.'''

Right now it's pretty fresh and only slowly getting any content at all.

Some in-progress ideas/concepts are:

*[[Alien Planet]] (game)
*[[Open Politics]] (game)
*[[Bayou]] (Page Info)
*[[RMD]] (Universal Communications)
*more to come...

Some concepts that have already been implemented:

*[[Tahoe]] (Data Manager)
*[[Jökulsárlón]] (Download Manager)
*[[Lantea]] (mapping/tracking)

Lantea

2011-11-28T12:48:08Z

KaiRo:

[http://stargate.wikia.com/wiki/Lantea Lantea] is a fictional ([http://en.wikipedia.org/wiki/Stargate_Atlantis Stargate: Atlantis]) mostly-ocean planet on which the Ancients erected the artificial "island" city of Atlantis.

The Lantea concept is a web app for mapping and tracking that is targeted to run on [https://wiki.mozilla.org/B2G B2G], but can run anywhere on modern browsers.

Base technologies:
* [http://wwww.openstreetmap.org/ OpenStreetMap] tiles - for rendering maps (should be parameterized so that alternative tiles like Google Maps/Satellite can also be used later on)
* [https://developer.mozilla.org/en/Using_geolocation Geolocation] - esp. watchPosition() for tracking positions
* [https://developer.mozilla.org/en/IndexedDB indexedDB] - for saving settings, current track, etc.
* [https://developer.mozilla.org/en/Using_Application_Cache appCache] - for locally caching resources so the app can be used offline
* [https://developer.mozilla.org/en/HTML/Canvas canvas] - for displaying tiles and painting position/tracks
* [http://en.wikipedia.org/wiki/GPS_eXchange_Format GPX format] - for downloading tracks / storing them externally

Resources:
* [http://concentriclivers.com/slippymap.html Canvas Slippy Map] - a simple canvas-based OpenStreetMap map display
* [https://github.com/dfacts/Slippy-Map-On-Canvas/#readme Slippy Map On Canvas] - a more elaborate canvas-based OpenStreetMap web app that can do e.g. touch events for panning and has zoom buttons
* [http://www.mardy.it/mappero Mappero] (a.k.a. [https://garage.maemo.org/projects/maemo-mapper/ maemo-mapper]) - a Maemo app that solved well what the Lantea app should provide on an HTML base
* [http://wiki.openstreetmap.org/wiki/OsmAnd OsmAnd] - an Android app that is similar

Lantea

2011-11-27T23:31:51Z

KaiRo:

[http://stargate.wikia.com/wiki/Lantea Lantea] is a fictional ([http://en.wikipedia.org/wiki/Stargate_Atlantis Stargate: Atlantis]) mostly-ocean planet on which the Ancients erected the artificial "island" city of Atlantis.

The Lantea concept is a web app for mapping and tracking that is targeted to run on [https://wiki.mozilla.org/B2G B2G], but can run anywhere on modern browsers.

Base technologies:
* [http://wwww.openstreetmap.org/ OpenStreetMap] tiles - for rendering maps (should be parameterized so that alternative tiles like Google Maps/Satellite can also be used later on)
* [https://developer.mozilla.org/en/Using_geolocation Geolocation] - esp. watchPosition() for tracking positions
* [https://developer.mozilla.org/en/IndexedDB indexedDB] - for saving settings, current track, etc.
* [https://developer.mozilla.org/en/Using_Application_Cache appCache] - for locally caching resources so the app can be used offline
* [https://developer.mozilla.org/en/HTML/Canvas canvas] - for displaying tiles and painting position/tracks
* [http://en.wikipedia.org/wiki/GPS_eXchange_Format GPX format] - for downloading tracks / storing them externally

Resources:
* [http://concentriclivers.com/slippymap.html CanvasS Slippy Map] - a simple canvas-based OpenStreetMap map display
* [https://github.com/dfacts/Slippy-Map-On-Canvas/#readme Slippy Map On Canvas] - a more elaborate canvas-based OpenStreetMap web app that can do e.g. touch events for panning and has zoom buttons
* [http://www.mardy.it/mappero Mappero] (a.k.a. [https://garage.maemo.org/projects/maemo-mapper/ maemo-mapper]) - a Maemo app that solved well what the Lantea app should provide on an HTML base
* [http://wiki.openstreetmap.org/wiki/OsmAnd OsmAnd] - an Android app that is similar

Lantea

2011-11-27T23:25:59Z

KaiRo: Created page with "[http://stargate.wikia.com/wiki/Lantea Lantea] is a fictional ([http://en.wikipedia.org/wiki/Stargate_Atlantis Stargate: Atlantis]) mostly-ocean planet on which the Ancients erec..."

[http://stargate.wikia.com/wiki/Lantea Lantea] is a fictional ([http://en.wikipedia.org/wiki/Stargate_Atlantis Stargate: Atlantis]) mostly-ocean planet on which the Ancients erected the artificial "island" city of Atlantis.

The Lantea concept is a web app for mapping and tracking that is targeted to run on [https://wiki.mozilla.org/B2G B2G], but can run anywhere on modern browsers.

Base technologies:
* [http://wwww.openstreetmap.org/ OpenStreetMap] tiles - for rendering maps (should be parameterized so that alternative tiles like Google Maps/Satellite can also be used later on)
* [https://developer.mozilla.org/en/Using_geolocation Geolocation] ([http://www.w3.org/TR/geolocation-API/ W3C standard) - esp. watchPosition() for tracking positions
* [https://developer.mozilla.org/en/IndexedDB indexedDB] - for saving settings, current track, etc.
* [https://developer.mozilla.org/en/Using_Application_Cache appCache] - for locally caching resources so the app can be used offline
* [https://developer.mozilla.org/en/HTML/Canvas canvas] - for displaying tiles and painting position/tracks
* [http://en.wikipedia.org/wiki/GPS_eXchange_Format GPX format] - for downloading tracks / storing them externally

Resources:
* [http://concentriclivers.com/slippymap.html CanvasS Slippy Map] - a simple canvas-based OpenStreetMap map display
* [https://github.com/dfacts/Slippy-Map-On-Canvas/#readme Slippy Map On Canvas] - a more elaborate canvas-based OpenStreetMap web app that can do e.g. touch events for panning and has zoom buttons
* [http://www.mardy.it/mappero Mappero] (a.k.a. [https://garage.maemo.org/projects/maemo-mapper/ maemo-mapper]) - a Maemo app that solved well what the Lantea app should provide on an HTML base
* [http://wiki.openstreetmap.org/wiki/OsmAnd OsmAnd] - an Android app that is similar

Main Page

2011-11-27T23:04:38Z

KaiRo:

'''This wiki is a random collection of ideas and concepts by Robert Kaiser.'''

Right now it's pretty fresh and only slowly getting any content at all.

Some in-progress ideas/concepts are:

*[[Alien Planet]] (game)
*[[Open Politics]] (game)
*[[Bayou]] (Page Info)
*[[RMD]] (Universal Communications)
*[[Lantea]] (mapping/tracking)
*more to come...

Some concepts that have already been implemented:

*[[Tahoe]] (Data Manager)
*[[Jökulsárlón]] (Download Manager)

Veridian 3

2011-03-16T18:20:03Z

KaiRo: /* Bugzilla Tags */

Veridian 3

2011-03-16T18:19:39Z

KaiRo: /* Prioritization Comments From Socorro Users */

Veridian 3

2011-03-15T21:02:26Z

KaiRo: /* Prioritization Comments From Socorro Users */

Veridian 3

2011-03-10T21:28:39Z

KaiRo: /* Prioritization Comments From Socorro Users */

Veridian 3

2011-03-09T20:06:06Z

KaiRo: /* Reports */

Veridian 3

2011-03-09T19:55:46Z

KaiRo: /* Explosive Crashes */

Veridian 3

2011-03-09T19:42:15Z

KaiRo: /* Bugzilla Tags */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Moved to http://wiki.mozilla.org/CrashKill/Plan/BugLists

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

Josh Matthews
I, like Jeff, would appreciate summaries of the data available - most recent 10 unique build ids, list of unique OS versions, range of uptimes, etc.
I would also be really interested in data about spikes - seeing a graph of the number of crashes for a particular signature over time would be useful to track trends.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

==== Problems with this proposal ====
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

==== Upsides of this proposal ====
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

==== Examples ====
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

A larger number of examples, including dist clamping (but no ADU) is available as an [[Media:Explosive_calc.ods‎|ODF spreadsheet]] ([[Media:Explosive_calc.pdf‎|PDF version]])

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-04T18:29:54Z

KaiRo: /* Reports */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (31): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (31): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (17): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (22): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (103): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (14): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (123): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (20): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

Josh Matthews
I, like Jeff, would appreciate summaries of the data available - most recent 10 unique build ids, list of unique OS versions, range of uptimes, etc.
I would also be really interested in data about spikes - seeing a graph of the number of crashes for a particular signature over time would be useful to track trends.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

==== Problems with this proposal ====
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

==== Upsides of this proposal ====
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

==== Examples ====
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

A larger number of examples, including dist clamping (but no ADU) is available as an [[Media:Explosive_calc.ods‎|ODF spreadsheet]] ([[Media:Explosive_calc.pdf‎|PDF version]])

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-04T14:54:08Z

KaiRo: /* Examples */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (31): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (31): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (17): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (22): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (103): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (14): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (123): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (20): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

Josh Matthews
I, like Jeff, would appreciate summaries of the data available - most recent 10 unique build ids, list of unique OS versions, range of uptimes, etc.
I would also be really interested in data about spikes - seeing a graph of the number of crashes for a particular signature over time would be useful to track trends.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

==== Problems with this proposal ====
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

==== Upsides of this proposal ====
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

==== Examples ====
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

A larger number of examples, including dist clamping (but no ADU) is available as an [[Media:Explosive_calc.ods‎|ODF spreadsheet]] ([[Media:Explosive_calc.pdf‎|PDF version]])

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-04T14:53:42Z

KaiRo: /* Examples */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (31): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (31): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (17): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (22): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (103): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (14): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (123): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (20): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

Josh Matthews
I, like Jeff, would appreciate summaries of the data available - most recent 10 unique build ids, list of unique OS versions, range of uptimes, etc.
I would also be really interested in data about spikes - seeing a graph of the number of crashes for a particular signature over time would be useful to track trends.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

==== Problems with this proposal ====
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

==== Upsides of this proposal ====
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

==== Examples ====
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

A larger number of examples, including dist clamping (but no ADU) is available as an [[Media:Explosive_calc.ods‎|ODF spreadsheet]] ([[:File:Explosive_calc.pdf‎|PDF version]])

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-04T14:51:46Z

KaiRo: /* Examples */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (31): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (31): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (17): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (22): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (103): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (14): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (123): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (20): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

Josh Matthews
I, like Jeff, would appreciate summaries of the data available - most recent 10 unique build ids, list of unique OS versions, range of uptimes, etc.
I would also be really interested in data about spikes - seeing a graph of the number of crashes for a particular signature over time would be useful to track trends.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

==== Problems with this proposal ====
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

==== Upsides of this proposal ====
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

==== Examples ====
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

A larger number of examples, including dist clamping (but no ADU) is available as an [[File:Explosive_calc.ods‎||ODF spreadsheet]] ([[File:Explosive_calc.pdf‎||PDF version]])

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-04T14:50:22Z

KaiRo: /* Examples */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (31): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (31): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (17): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (22): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (103): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (14): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (123): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (20): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

Josh Matthews
I, like Jeff, would appreciate summaries of the data available - most recent 10 unique build ids, list of unique OS versions, range of uptimes, etc.
I would also be really interested in data about spikes - seeing a graph of the number of crashes for a particular signature over time would be useful to track trends.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

==== Problems with this proposal ====
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

==== Upsides of this proposal ====
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

==== Examples ====
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

A larger number of examples, including dist clamping (but no ADU) is available as an [[File:Explosive_calc.ods‎|ODF spreadsheet]] ([[File:Explosive_calc.pdf‎|PDF version]])

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-04T14:49:14Z

KaiRo: /* Examples */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (31): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (31): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (17): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (22): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (103): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (14): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (123): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (20): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

Josh Matthews
I, like Jeff, would appreciate summaries of the data available - most recent 10 unique build ids, list of unique OS versions, range of uptimes, etc.
I would also be really interested in data about spikes - seeing a graph of the number of crashes for a particular signature over time would be useful to track trends.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

==== Problems with this proposal ====
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

==== Upsides of this proposal ====
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

==== Examples ====
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

A larger number of examples, including dist clamping (but no ADU) is in [[File:Explosive_calc.ods‎]] (PDF version: [[File:Explosive_calc.pdf‎]])

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

File:Explosive calc.pdf

2011-03-04T14:48:32Z

KaiRo: PDF version of explosiveness sheet

PDF version of explosiveness sheet

File:Explosive calc.ods

2011-03-04T14:46:49Z

KaiRo: uploaded a new version of "File:Explosive calc.ods": Add more examples, adjust variables to be more sensitive

Spread sheet of examples for the explosivenness calculation

Veridian 3

2011-03-04T14:45:38Z

KaiRo: /* Criteria Proposal */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (31): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (31): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (17): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (22): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (103): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (14): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (123): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (20): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

Josh Matthews
I, like Jeff, would appreciate summaries of the data available - most recent 10 unique build ids, list of unique OS versions, range of uptimes, etc.
I would also be really interested in data about spikes - seeing a graph of the number of crashes for a particular signature over time would be useful to track trends.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

==== Problems with this proposal ====
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

==== Upsides of this proposal ====
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

==== Examples ====
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

A larger number of examples, including dist clamping (but no ADU) is in [[File:Explosive_calc.ods‎]]

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-04T01:36:52Z

KaiRo: /* Prioritization Comments From Socorro Users */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (31): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (31): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (17): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (22): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (103): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (14): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (123): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (20): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

Josh Matthews
I, like Jeff, would appreciate summaries of the data available - most recent 10 unique build ids, list of unique OS versions, range of uptimes, etc.
I would also be really interested in data about spikes - seeing a graph of the number of crashes for a particular signature over time would be useful to track trends.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

A larger number of examples, including dist clamping (but no ADU) is in [[File:Explosive_calc.ods‎]]

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-03T22:02:28Z

KaiRo: /* Bugzilla Tags */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (31): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (31): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (17): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (22): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (103): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (14): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (123): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (20): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

A larger number of examples, including dist clamping (but no ADU) is in [[File:Explosive_calc.ods‎]]

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-03T18:42:31Z

KaiRo: /* Criteria Proposal */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

A larger number of examples, including dist clamping (but no ADU) is in [[File:Explosive_calc.ods‎]]

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

File:Explosive calc.ods

2011-03-03T18:40:20Z

KaiRo: Spread sheet of examples for the explosivenness calculation

Spread sheet of examples for the explosivenness calculation

Veridian 3

2011-03-03T18:26:50Z

KaiRo: /* Criteria Proposal */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".
[[File:explosice_calc.ods]]

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-03T17:22:52Z

KaiRo: /* Criteria Proposal */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (20 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-03T16:59:30Z

KaiRo: /* Criteria Proposal */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# For each set, calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (100 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# For each set, calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-03T16:53:57Z

KaiRo: /* Reports */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# Calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (100 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# Calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/
* Top second frame: http://people.mozilla.com/~aphadke/topSecondFrame/

Veridian 3

2011-03-02T23:15:47Z

KaiRo: /* Prioritization Comments From Socorro Users */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is:
How many other users who have this crash also running Firebug?
If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# Calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (100 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# Calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/

Veridian 3

2011-03-02T23:15:05Z

KaiRo: /* Prioritization Comments From Socorro Users */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is: How many other users who have this crash also running Firebug? If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

Jeff Muizelaar
It would be nice if it was possible to get more summary information about a crash.
For example: What build ids does this crash all occur with? What operating system versions does this all occur with? etc.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# Calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (100 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# Calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/

Veridian 3

2011-03-02T12:49:13Z

KaiRo: /* Prioritization Comments From Socorro Users */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

johnjbarton
Of course *all* of my crashes involve Firebug. The number one question I have when I visit crash-stats site is: How many other users who have this crash also running Firebug? If the answer is 95%, then I better spend some time on it because no one else will. If the answer is 5%, I'm having lunch.

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# Calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (100 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# Calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/

Veridian 3

2011-03-02T01:47:35Z

KaiRo: /* Prioritization Comments From Socorro Users */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

https://bugzilla.mozilla.org/show_bug.cgi?id=551669 provide graphs by crash date too
Smokey Ardisson (way behind; no bugmail - do not email) <alqahira@ardisson.org> changed:
CC| |kairo@kairo.at

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# Calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (100 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# Calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/

Veridian 3

2011-03-01T22:10:56Z

KaiRo: /* Criteria Proposal */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# Calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (100 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# Calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering the warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/

Veridian 3

2011-03-01T22:10:19Z

KaiRo: /* Criteria Proposal */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# Calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (100 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# Calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering a warning.
** On later days, should have triggered easily on both values, even with clamping of "dist".

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/

Veridian 3

2011-03-01T22:09:33Z

KaiRo: /* Criteria Proposal */

On a planet called [http://memory-alpha.org/wiki/Veridian_III Veridian III], a decisive battle was fought to prevent a future firing of a rocket into a star that would change gravitational forces and make "the nexus" crash into the planet as well as destroy the planet with a shock wave. Preventing this catastrophy made the crash of the USS Enterprise on the planet controllable as to suffer no human casualties.

In the same spirit, the project I internally dub "Veridian 3" is about dealing with crashes to make the bad ones preventable and other ones more controllable, all through prioritizing Socorro work.

== Project areas ==

Those areas have been specified in the contract:

* Improving Crash Data Integrity
** Identification and Removal of Duplicate Crash reports
* Improving Search Capabilities
* Improving Classification and Characterization of Crash Reports and Improved Signature Generation
* additional correlation reports to help identifying circumstances around the crash and steps to reproduce
* Improve Trend Reports to identify and alert teams about Explosive bugs

== Bugzilla Tags ==

Those are used for the classification of Socorro bugs, all starting with "V3", and those will be documented here. In brackets, there are bug counts as of 02/23.

* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-integrity&sharer_id=5189 V3-integrity]''' (30): Affecting Crash Data Integrity, i.e. quality of the original data we have stored
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-search&sharer_id=5189 V3-search]''' (28): Search Capabilities
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-classify&sharer_id=5189 V3-classify]''' (47): Classification and Characterization of Crash Reports and Signature Generation
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-correlation&sharer_id=5189 V3-correlation]''' (33): Correlation reports to help identifying circumstances around the crash and steps to reproduce
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-trends&sharer_id=5189 V3-trends]''' (15): Trend Reports, e.g. to identify and alert teams about Explosive bugs
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-newreports&sharer_id=5189 V3-newreports]''' (18): New reports (requests for generating new reports)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UI&sharer_id=5189 V3-UI]''' (102): User Interface issues
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-UItweaks&sharer_id=5189 V3-UItweaks]''' (58): UI tweaks (probably easy to solve, small UI issues) - subgroup of V3-UI
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-nonHTMLoutput&sharer_id=5189 V3-nonHTMLoutput]''' (12): Non-HTML/web output (.csv, feeds, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-notify&sharer_id=5189 V3-notify]''' (13): Notifications (to be) sent out by the Socorro system
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-infra&sharer_id=5189 V3-infra]''' (120): Infrastructure and backend issues (note: out of the direct focus of my project, subject to internal planning in the Socorro team)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-config&sharer_id=5189 V3-config]''' (21): Configuration adaptations (skiplist additions, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-productization&sharer_id=5189 V3-productization]''' (14): Making Socorro a product that can be deployed and understood by others (documentation, etc.)
* '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=V3-datarequest&sharer_id=5189 V3-datarequest]''' (6): Data requests (bugs that request data through manual jobs)

== Prioritization Comments From Socorro Users ==

<wsmwk> KaiRo: second tier needs might be bug 421119, bug 518823, bug 578376, bug 411354. third tier: bug 527304, bug 512910, better workflow for updating skiplist)
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=421119 min, P3, 2.1, nobody, NEW, function for socorro to compare stacks of two or more crash reports
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=518823 enh, --, Future, nobody, NEW, indicate bug's status for bugzilla keyword topcrash
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=578376 nor, --, ---, nobody, NEW, multiple crashes from a single person should have less weight then many crashes from different peopl
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=411354 nor, P1, 2.0, nobody, REOP, Add ability to search by build ID
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=527304 enh, --, ---, nobody, REOP, provide smart analysis ala talkback
<firebot> Bug https://bugzilla.mozilla.org/show_bug.cgi?id=512910 enh, --, ---, nobody, NEW, Make it easier to analyze crashes that share a signature

== Explosive Crashes ==

Notes on the work on a set of criteria for finding explosive crash reports - [https://bugzilla.mozilla.org/show_bug.cgi?id=629049 bug 629049] is the tracker bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=629062 bug 629062] is detection. The [https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking PRD doc] has some surrounding info, but no criteria yet.

=== Personal Notes ===

* Sharp/significant increase at certain wall-clock time across versions
* Sharp/significant increase at certain build ID (date?) on single version/series (possibly ignoring everything in version string starting with first letter if the version ends in "pre", to have e.g. 5.0a3pre->5.0b1pre or 4.0b11pre->4.0b12pre not disturb the analysis)
* Ignore (suspected) duplicates
* Frequency weighted by ADU more important than bare count (from something chofmann has said)
* I'm not fond of topcrash rank comparisons, as 20 crashes with similar frequency changing place looks overvalued there, while e.g. #1 having 10,000 crashes and #3 having 500 fully mask #2 exploding from 600 to 5,000 in a day.

=== Criteria Proposal ===

This is a quite rough proposal right now.

# Get two sets of numbers per signature:
#* non-duplicate crashes occurred per day and total ADU for the last 10 days
#* non-duplicate crashes and ADU per combination of version series (see personal notes) and date of build ID, for the last 10 available build ID dates in the version series
# Calculate (if there are at least 4 values in the set):
#* average crashes per ADU over 7 values before recent value ("base")
#* average ADU over those values ("avgADU")
#* distance of that average to the highest value in set ("dist"), clamped to a minimum of (100 crashes/avgADU)
#* recent value per ADU ("data")
#* '''(total|version)_explosiveness_1''' = (data-base)/dist
# Calculate (if there are at least 6 values in the set):
#* average crashes per ADU over 7 values before recent 3 values ("base")
#* average ADU over those values ("avgADU")
#* standard deviation of that average ("dist"), clamped to a minimum of (50 crashes/avgADU)
#* average of recent 3 values per ADU ("data")
#* '''(total|version)_explosiveness_3''' = (data-base)/dist
# Mark as explosive in UI if '''*_explosiveness_1 > 3''' or '''*_explosiveness_3 > 2'''.

Problems with this proposal:
* Completely arbitrary numbers for explosiveness marking limits and "dist"-clamping, need to see if they catch all explosives and/or catch too much.
* If there's no large enough set of numbers to work with, there's no useful explosiveness.
* It's unclear if the version-based numbers give really useful additional value, they also create a multitude of explosiveness numbers to store (2 per version series).
* There might be an argument for only calculating the second (*_explosiveness_3) measure, as it's fine-grained enough to catch highly explosive crashes on the first day of explosion.

Upsides of this proposal:
* Recognizes that dupes and ADU changes can make base values fluctuate and gets rid of those problems.
* The clamping of "dist" doesn't just prohibit divisions by zero, but also deals potential skew due to tiny fluctuations in small numbers.
* Having explosiveness numbers available to UI enables flexibility in marking, sorting and changing limits.

Example:
* Bug 554660 (see below) has an interesting example of numbers to look at for this: totals of 54, 72, 86, 83, 67, 46, 47, 123, 131 for 2010-03-08 through 2010-03-16. Here's a look at how this algorithm does, ignoring ADU, which are not given there, and therefore also the clamping:
** On 2010-03-15, total_explosiveness_1 would have been 2.7, not yet triggering (?), and total_explosiveness_3 would have been slightly negative, also not triggering.
** On 2010-03-16, total_explosiveness_1 would have been 1.2, not triggering, but total_explosiveness_3 would have been 2.2, triggering a warning.
** On later days, should have triggered easily on both values.

=== User Comments ===

From https://wiki.mozilla.org/Socorro:PRD_Interviews

damon:
* (initial) growth of more than 25 positions in the ranking
* upwards change in rank and no related bugzilla id
* time since startup < 1 minute
* highlight these crashes in red or something

From https://bugzilla.mozilla.org/show_bug.cgi?id=525316

morgamic:
My suggestion for a delta to watch is an increase in crash frequency of more
than 50-75% and new crashes in the top 20 overall signatures by version.

=== Data From Previous Explosive Crash Bugs ===

Used the [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=explos explosive bug query] to find those, trying to pull info out on how those were explosive.

* https://bugzilla.mozilla.org/show_bug.cgi?id=503946
** #16 tc (2-week) for 3.6 on 2010-01-26, #2 in 1-day, #3 in 3-day
** crash numbers with that signature: 2010-01-24 145 (days before similar), 2010-01-25 1950, 2010-01-26 11731
** percentage of total crashes on 2010-01-25 was 4 times as high on 3.6 as on 3.5
* https://bugzilla.mozilla.org/show_bug.cgi?id=528798
** Rise from 7-18 null signature crashes with comments to >100 (with some days of 30s or 50s) within a week or less after the 3.5.5 release.
** Increase in total null-signature crashes from <5000 to >6000 within 2-3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=530074
** crashes with that signature jumped from 45-65 to 570 and higher within 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=536974
** jumped up 41 ranks in top crasher analysis tp #15 on 3.6b5 in 3 days
** two signatures, both from <30 total crashes per day to >300 within a week
* https://bugzilla.mozilla.org/show_bug.cgi?id=538687
** uptick from 45-60 (2009-12-16 to 2010-01-02, with some single-day spikes above) to multiple consecutive days with 80-100 (2010-01-03 to -07) total crashes per day
** percentage of total crashes on 3.6 is about a factor 50-100 higher than on 3.5
** up to 416 crashes on 2010-03-10, 600-900 on 2010-03-12 to -14, >1600 on 2010-03-15, >1000 until -19
** #45 topcrash in 3.6b5 (2010-01-08), #8 in early 3.6.2 top crash data (2010-03-23)
* https://bugzilla.mozilla.org/show_bug.cgi?id=538998
** From 0 to >100 in two days, from 94-115 to >3700 in one day
** #10 tc in early 3.6rc1 reports
** from 3-12 total crashes per hour to 100-600 with a sharp cutoff hour
* https://bugzilla.mozilla.org/show_bug.cgi?id=543646
** from 0 to >1000 crashes in two days, staying there at least 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=546632
** new crash in top 30 tc on 3.6
** from 0 to >100 in a day, from 260 to >1000 in 7 days
** roughly factor 5 between 3.6 and other versions in percentage of total crashes
* https://bugzilla.mozilla.org/show_bug.cgi?id=547210
** 0 to >3000 in a day, stayed >1000 for 3 days at least
* https://bugzilla.mozilla.org/show_bug.cgi?id=547622
** Top 50 Crash for 3.6 (+149!) Firefox 3.6 Crash Report
** started showing up around the first of November with 1-10 crashes per day, then 10-40 crashes per day in Dec, 100-150 in January, and last few days of February was running at 400-712 crashes per day
* https://bugzilla.mozilla.org/show_bug.cgi?id=553581
** +224 positions up to Top 33 Crash for 3.6
* https://bugzilla.mozilla.org/show_bug.cgi?id=554660
** from 45-90 to >100 in a day, >400 in 3 days and rising further (>600 in 6 days etc.)
* https://bugzilla.mozilla.org/show_bug.cgi?id=558955
** 200-500 crashes per day in first half of April '10, up from 0-5 crashes per day in Nov '09 through early March '10
** ~5 to >100 in 3 days, somewhat back down, then ~80 to >200 in 3 days, 150-180 to >500 in 3 days
* https://bugzilla.mozilla.org/show_bug.cgi?id=570722
** from 7-18 to >600 in a day
* https://bugzilla.mozilla.org/show_bug.cgi?id=595957
** from 30-90 to >3000 in 3 days

== Reports ==

Some tools and reports on crash data are currently outside the main Socorro systems:
* List of Firefox nightly builds, liked to crash queries per build: http://dbaron.org/mozilla/crashes-by-build
* Top crashes per day: http://people.mozilla.org/~chofmann/crash-stats/
* Correlation analysis (CPU cores, modules, add-ons): http://people.mozilla.com/crash_analysis/20110228/