1 2 Previous Next 25 Replies Latest reply on Dec 3, 2012 6:59 AM by jlin

    Best way to check data entry reliability between 2 people

    jlin

      Title

      Best way to check data entry reliability between 2 people

      Post

           Hello,

           I am a Filemaker beginner using FMP 12 to code (or enter) data for quantitative research.  I have one coding partner who will code the same items, for the purpose of assessing our agreement on these items and computing statistical reliability.  What is the best way to do this in FMP?  I initially thought of making a set of duplicate fields just for the coder, but this seems messy.   

           Pleate note that my coder does not have FMP and is going to code using Instant Web Publishing, though if necessary I will get another license.  

           Any help would be appreciated! 

           Thank you,

           Joy

        • 1. Re: Best way to check data entry reliability between 2 people
          philmodjunk

               Can you explain what you mean by "computing statistical reliability"? in more detail?

               You'll find that you have more interface design options available to you if you don't use Instant Web Publishing, BTW.

          • 2. Re: Best way to check data entry reliability between 2 people
            jlin

                 Sorry for the confusion.  Please disregard the part about computing reliaiblity, as I will be doing the actual computing on a different program.  I was just trying to provide some context.

                 To illustrate, let's say I have a "survey" layout.  I need a way for 2 people to fill out the survey, and then to compare the values they enter for each item.  So on Item 1,  I may have a value list of "High," "Medium," and "Low," and I may code the item as "High."  My coder will also enter a value for this item, which may or may not be the same as what I enter.  Ultimately, I need to compare 2 sets of codes for each field on my survey.  Is there a way to do this that doesn't involve making duplicate fields- one for me, and one for the coder? 

                 In regards to IWP, I was told that it should work fine for basic data entry.  (If you have additional insight on this, please let me know.)  I am trying it out for now, hoping that I will not need to purchase another license.

                 Hope this is clear.

                 Thank you.

            • 3. Re: Best way to check data entry reliability between 2 people
              philmodjunk

                   Use different records for each person filling in responses to your survey questions. You can then compare the records instead of the duplicate fields. This makes for a much simpler design and allows you to compare responses for a flexible number of users should that option be needed.

                   You may find this thread on designing a survey database in FileMaker helpful:

              Need aid on generating a report from a survey layout.

              • 4. Re: Best way to check data entry reliability between 2 people
                jlin

                     Actually, this is not a true survey filled out by multiple people.  Each record in my database is an empirical study, and my coder and I are coding various aspects of the study.  The goal is for us to code them nearly exactly the same way.    

                • 5. Re: Best way to check data entry reliability between 2 people
                  philmodjunk

                       That doesn't alter my advice to use one record for each coder's reponses to each question. Your study still, from what little I can see in your posts here, has a lot in common with a survery or questionaire. It seems that two people are posting repsponses to the same list of questions--and that should lend itself quite handily to the database design outlined in the recommended thread.

                  • 6. Re: Best way to check data entry reliability between 2 people
                    jlin

                         Thank you for your advice and for the reference.  

                         If I am understanding things correctly, I don't think this database design would be appropriate for my research, primarily because what is of interest in my project is not the variation in responses but the variation in the studies.  The goal is to have one master set of "responses" to each item about every study, and then use this as a dataset for analysis.   (Having a second coder is just a means to avoid bias in the primary coder (me), whose responses are the only ones that matter.)

                         For further background, this is a skeleton of my ERD:  Studies --> Measures --> Effect Sizes.  The effect sizes are what I will eventually be analyzing, and most of the items on my coding form are based on Effect Size table, but I also have Study-level and Measure-level codes.

                         My coder just needs to code about 20% of all the studies in the database, so I just need a way to compare our codes for that 20%.  Would creating duplicate codes on a separate layout be the only option that doesn't involve restructuring my database?  And is there any real problem with doing it this way?

                         My apologies for not explaining this very well.

                    • 7. Re: Best way to check data entry reliability between 2 people
                      philmodjunk

                           However, you describe it. If you are going to compare data entered by two different users to the same record in Measures, using separate records will be much more flexible than using parallel fields in the same record.

                           It's better in the long run to have a well designed data model (set of tables and relationships) that allows you to work effectively with your data than to try to somehow get by with a less than optimum design.

                            

                      • 8. Re: Best way to check data entry reliability between 2 people
                        jlin

                             I tried to draw out an ERD based on this input, but am having some trouble.   How do I deal with the fact that my coding form has items based on multiple levels (Studies --< Measures --< Effect Sizes)?   

                             I have 2 layouts for the coding form: The first (Page 1) is study-level and includes a portal to enter data about Measures.  The second layout (Page 2) is based on effect sizes.  

                        • 9. Re: Best way to check data entry reliability between 2 people
                          philmodjunk

                               Studies --< Measures --< Effect Sizes

                               is a very simple outline of an ERD, but without the match fields specified.

                               You have a number of options for synchronizing page 2 with Page 1. WHich is best may well depend on your data entry needs.

                               It can be a layout based on Measures with a portal to Effect Sizes, for example. A button on Page 1 can pull up a found set of All Measures for that Sudy. You can flip through the records and see the Effect Sizes records for each measure record listed in a portal.

                               Go To Related Records is an option you can use with such a button to bring up the related Measures Records for a given study record.

                               And for reporting purposes, you can base a list view layout on Effect Sizes and include data from the other two tables in other layout parts such as headers, grand summary and sub summary parts.

                          • 10. Re: Best way to check data entry reliability between 2 people
                            jlin

                                 Thank you for your replies.  To clarify, the question I asked in my previous post about the coding form including multiple levels was in reference to the issue of designing the database so that 2 people can enter data for the same record.  In the survey example given in the referenced thread, where one record in the Respondants table represents each person filling out the survey, the Respondants and Survey Questions tables were connected by a Responses table.  I was trying to figure out how to restructure my database based on this example, but got hung up the fact that my "Responses" (i.e., the fields on my coding form) are based on multiple tables (Study, Measure, and Effect Size).  If I have something like this: Coder ---< Codes >--- Study, how do I incorporate those other tables?

                                 By the way, in the outline I gave earlier, Study --< Measure--< Effect Size, each table is linked by a serial ID number (Study: __pk_StudyID = Measure:_fk_StudyID and Measure:__pk_MeasureID = Effect Size: __fk_MeasureID).

                                 Also, would using a separate database file be useful?  I know you can use link your data to external files somehow but am not familiar with this feature.

                                  

                                  

                                  
                            • 11. Re: Best way to check data entry reliability between 2 people
                              philmodjunk
                                   

                                        To clarify, the question I asked in my previous post about the coding form including multiple levels was in reference to the issue of designing the database so that 2 people can enter data for the same record.

                                   And I am recommending that they not enter data into the same record, but two related records.

                                   I keep running into the classic limitations built into a forum like this--I only know what you choose to share in your posts. I don't have a clear enough picture of what two "coders" might need to enter. Can you sketch out an example of what they might need to enter?

                                   I have assumed the relationship details you just posted, but have also assumed that the only data your second coder has to enter is data in the Effect_Size table. I've assumed that the coder would select the appropriate Measure record and then enters related data in the Effect_Size table, but with each coder entering an ID into an added field so that you can see which Effect_Size records were created by each coder.

                              • 12. Re: Best way to check data entry reliability between 2 people
                                jlin

                                     This is all very much appreciated.  I'll give this another shot.  Here is the sketch you requested: 

                                     1) Let's say the database has 1 study.  This study is associated with 1 measure, and this measure is associated with 1 effect size (ES).

                                     2a) I have 2 layouts for entering information about this study: The first layout (Page 1) is at the Study level.  On this layout, there is a field indicating the research design used in this study.  This field is formatted with this value list: Experimental, Quasi-experimental, Single-group.  On this same layout, I also have a portal for adding measures, with fields about those measures.  Here, I have a field designating the measure type, with this value list: Researcher-constructed, Practiioner-constructed, Standardized.    

                                     2b) The second layout (Page 2) is at the ES level.  At the top of the layout, I have a foreign key field to choose a related measure.  This links the ES with the study.  (I'm sure there's a better way to do this, but this is what I have so far.)  The rest of the fields are all at the ES level.  All of the fields on Page 1 and 2 areformatted with value lists (or eventually will be).

                                     3) My coder has to enter the exact same information that I do, just in a way that allows my data to be intact, and that allows me to compare her data to mine. 

                                     I hope this is more clear.  Thank you.

                                • 13. Re: Best way to check data entry reliability between 2 people
                                  philmodjunk
                                       

                                            I'm sure there's a better way to do this

                                       That's actually a very commonly used layout design. I did describe an alternative that also works in one of my earlier posts but am not suggesting that it's better--it eliminates the need for selecting the measure, but also requires doing your data entry in a portal which isn't always the best approach.

                                       What you describe sounds consistent with my last post. Your second coder would access the database, select the Measure from your value list and then enters ES data into a new record, but enters a different ID into one field in the ES table so that you can see which records were created by a given user.

                                  • 14. Re: Best way to check data entry reliability between 2 people
                                    jlin

                                         1) I think I see what you're saying, but the second coder actually has to enter all of the data I described in points 2a and 2b, or data at the study, measure, and ES levels.  Given this, I don't know how I would "tag" the records by coder in the way that you're describing.  

                                    2) "Your second coder would access the database, select the Measure from your value list and then enters ES data into a new record, but enters a different ID into one field in the ES table so that you can see which records were created by a given user."

                                         In this case, would the ES table just have to be linked with a parent Coder table?

                                    1 2 Previous Next